[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:33:31 dist-test-slave-grlk krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:33:31 dist-test-slave-grlk krb5kdc[399](info): set up 2 sockets
Sep 27 02:33:31 dist-test-slave-grlk krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 02:33:31.896801 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/krb5kdc/test-user.keytab.
Sep 27 02:33:31 dist-test-slave-grlk krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:33:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:33:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:33:32.007977 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45855
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45855
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:33:32.111021 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:33:32.111267 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:33:32.111304 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:33:32.115273 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:33:32.115334 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:33:32.115351 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:33:32.115366 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:33:32.115381 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:33:32.120821 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:33:32.121835 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:33:32.122829 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:33:32.131412 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:32.131400 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:32.131649 412 server_base.cc:1034] running on GCE node
W20260927 02:33:32.131661 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:32.132165 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:33:32.133257 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:33:32.134394 412 hybrid_clock.cc:648] HybridClock initialized: now 1790476412134329 us; error 79 us; skew 500 ppm
Sep 27 02:33:32 dist-test-slave-grlk krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:33:32.137020 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:33:32.137838 412 webserver.cc:460] Webserver started at http://127.0.98.62:45141/ using document root <none> and password file <none>
I20260927 02:33:32.138345 412 fs_manager.cc:359] Metadata directory not provided
I20260927 02:33:32.138406 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:33:32.138593 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:33:32.140506 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "73b8364e6d82430db06d8e999402f62f"
format_stamp: "Formatted at 2026-09-27 02:33:32 on dist-test-slave-grlk"
server_key: "0f9ce70ad65aed6dde381903817cf2db"
server_key_iv: "098a8c23a4e86106d75e22e0e57b68f8"
server_key_version: "encryptionkey@0"
I20260927 02:33:32.141048 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "73b8364e6d82430db06d8e999402f62f"
format_stamp: "Formatted at 2026-09-27 02:33:32 on dist-test-slave-grlk"
server_key: "0f9ce70ad65aed6dde381903817cf2db"
server_key_iv: "098a8c23a4e86106d75e22e0e57b68f8"
server_key_version: "encryptionkey@0"
I20260927 02:33:32.144470 412 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:33:32.146625 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:32.147770 412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:33:32.147876 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "73b8364e6d82430db06d8e999402f62f"
format_stamp: "Formatted at 2026-09-27 02:33:32 on dist-test-slave-grlk"
server_key: "0f9ce70ad65aed6dde381903817cf2db"
server_key_iv: "098a8c23a4e86106d75e22e0e57b68f8"
server_key_version: "encryptionkey@0"
I20260927 02:33:32.147964 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:33:32.162352 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:33:32.165242 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:33:32.165408 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:33:32.171931 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45855
I20260927 02:33:32.171945 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45855 every 8 connection(s)
I20260927 02:33:32.172825 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:33:32.175477 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:33:32.180754 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73b8364e6d82430db06d8e999402f62f: Bootstrap starting.
I20260927 02:33:32.183226 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73b8364e6d82430db06d8e999402f62f: Neither blocks nor log segments found. Creating new log.
I20260927 02:33:32.183414 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 412
I20260927 02:33:32.183568 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:33:32.184099 480 log.cc:824] T 00000000000000000000000000000000 P 73b8364e6d82430db06d8e999402f62f: Log is configured to *not* fsync() on all Append() calls
I20260927 02:33:32.185087 392 external_mini_cluster.cc:1428] Setting key 25b6cd20fc70c747f4123329ab56d8f1
I20260927 02:33:32.186174 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73b8364e6d82430db06d8e999402f62f: No bootstrap required, opened a new log
I20260927 02:33:32.190320 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73b8364e6d82430db06d8e999402f62f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73b8364e6d82430db06d8e999402f62f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45855 } }
I20260927 02:33:32.190531 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73b8364e6d82430db06d8e999402f62f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:33:32.190575 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73b8364e6d82430db06d8e999402f62f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73b8364e6d82430db06d8e999402f62f, State: Initialized, Role: FOLLOWER
I20260927 02:33:32.190945 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73b8364e6d82430db06d8e999402f62f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73b8364e6d82430db06d8e999402f62f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45855 } }
I20260927 02:33:32.191052 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73b8364e6d82430db06d8e999402f62f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:33:32.191107 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73b8364e6d82430db06d8e999402f62f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:33:32.191195 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73b8364e6d82430db06d8e999402f62f [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:33:32.191960 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73b8364e6d82430db06d8e999402f62f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73b8364e6d82430db06d8e999402f62f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45855 } }
I20260927 02:33:32.192251 480 leader_election.cc:302] T 00000000000000000000000000000000 P 73b8364e6d82430db06d8e999402f62f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73b8364e6d82430db06d8e999402f62f; no voters:
I20260927 02:33:32.192544 480 leader_election.cc:288] T 00000000000000000000000000000000 P 73b8364e6d82430db06d8e999402f62f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:33:32.192613 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73b8364e6d82430db06d8e999402f62f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:33:32.192836 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73b8364e6d82430db06d8e999402f62f [term 1 LEADER]: Becoming Leader. State: Replica: 73b8364e6d82430db06d8e999402f62f, State: Running, Role: LEADER
I20260927 02:33:32.193171 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73b8364e6d82430db06d8e999402f62f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73b8364e6d82430db06d8e999402f62f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45855 } }
I20260927 02:33:32.193760 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73b8364e6d82430db06d8e999402f62f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:33:32.196496 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73b8364e6d82430db06d8e999402f62f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73b8364e6d82430db06d8e999402f62f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73b8364e6d82430db06d8e999402f62f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45855 } } }
I20260927 02:33:32.196552 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73b8364e6d82430db06d8e999402f62f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73b8364e6d82430db06d8e999402f62f. Latest consensus state: current_term: 1 leader_uuid: "73b8364e6d82430db06d8e999402f62f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73b8364e6d82430db06d8e999402f62f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45855 } } }
I20260927 02:33:32.196641 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73b8364e6d82430db06d8e999402f62f [sys.catalog]: This master's current role is: LEADER
I20260927 02:33:32.196651 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73b8364e6d82430db06d8e999402f62f [sys.catalog]: This master's current role is: LEADER
W20260927 02:33:32.198776 499 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 73b8364e6d82430db06d8e999402f62f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:33:32.198853 499 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:33:32.198918 501 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:33:32.199566 501 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 02:33:32 dist-test-slave-grlk krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:33:32.205196 501 catalog_manager.cc:1347] Generated new cluster ID: e8563a667b0b4f29bfa8cf7f2545a8d8
I20260927 02:33:32.205253 501 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:33:32.208900 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:32.188313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52950 (local address 127.0.98.62:45855)
0927 02:33:32.188756 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:32.188775 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:32.189894 (+ 1119us) server_negotiation.cc:407] Connection header received
0927 02:33:32.192530 (+ 2636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:32.192547 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:32.192926 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:32.193291 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:32.195956 (+ 2665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:32.197370 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:32.198960 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:32.199444 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:32.199601 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:32.203855 (+ 4254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:32.203861 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:32.203876 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:32.203905 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:32.205851 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:32.206327 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:32.206332 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:32.206336 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:32.206445 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:32.206762 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:32.206766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:32.206768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:32.207147 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:32.207328 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:32.207854 (+ 526us) server_negotiation.cc:299] Negotiation successful
0927 02:33:32.208102 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":136,"threads_started":1}
I20260927 02:33:32.225742 501 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:33:32.226611 501 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:33:32.234249 501 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73b8364e6d82430db06d8e999402f62f: Generated new TSK 0
I20260927 02:33:32.234835 501 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:33:32.265671 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45855
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:33:32.369997 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:33:32.370235 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:33:32.370277 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:33:32.374168 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:33:32.374217 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:33:32.374272 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:33:32.379276 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45855
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:33:32.380263 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:33:32.381254 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:33:32.390744 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:32.390744 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:32.390879 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:32.391342 506 server_base.cc:1034] running on GCE node
I20260927 02:33:32.391780 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:33:32.392455 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:33:32.393643 506 hybrid_clock.cc:648] HybridClock initialized: now 1790476412393622 us; error 36 us; skew 500 ppm
Sep 27 02:33:32 dist-test-slave-grlk krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:33:32.396379 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:33:32.397269 506 webserver.cc:460] Webserver started at http://127.0.98.1:39551/ using document root <none> and password file <none>
I20260927 02:33:32.397787 506 fs_manager.cc:359] Metadata directory not provided
I20260927 02:33:32.397850 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:33:32.398053 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:33:32.400048 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/ts-0/data/instance:
uuid: "cf03610db13a4a3d9533f5858ea439d9"
format_stamp: "Formatted at 2026-09-27 02:33:32 on dist-test-slave-grlk"
server_key: "c510d490fba8d87074baf9329d986ca1"
server_key_iv: "df44606da59e7647921765c8158e9b36"
server_key_version: "encryptionkey@0"
I20260927 02:33:32.400573 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cf03610db13a4a3d9533f5858ea439d9"
format_stamp: "Formatted at 2026-09-27 02:33:32 on dist-test-slave-grlk"
server_key: "c510d490fba8d87074baf9329d986ca1"
server_key_iv: "df44606da59e7647921765c8158e9b36"
server_key_version: "encryptionkey@0"
I20260927 02:33:32.404286 506 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 02:33:32.406575 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:32.407698 506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:33:32.407796 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/ts-0/wal
uuid: "cf03610db13a4a3d9533f5858ea439d9"
format_stamp: "Formatted at 2026-09-27 02:33:32 on dist-test-slave-grlk"
server_key: "c510d490fba8d87074baf9329d986ca1"
server_key_iv: "df44606da59e7647921765c8158e9b36"
server_key_version: "encryptionkey@0"
I20260927 02:33:32.407881 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:33:32.426851 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:33:32.429821 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:33:32.429986 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:33:32.430544 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:33:32.431313 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:33:32.431356 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:32.431406 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:33:32.431433 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:32.440344 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44369
I20260927 02:33:32.440353 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44369 every 8 connection(s)
I20260927 02:33:32.441254 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 02:33:32 dist-test-slave-grlk krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:33:32.451375 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 506
I20260927 02:33:32.451481 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:33:32.451743 392 external_mini_cluster.cc:1428] Setting key ef3afebad182f25a5e90d318b7b2468b
I20260927 02:33:32.454682 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:32.443399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59839 (local address 127.0.98.62:45855)
0927 02:33:32.443558 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:32.443564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:32.444392 (+ 828us) server_negotiation.cc:407] Connection header received
0927 02:33:32.445681 (+ 1289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:32.445684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:32.445735 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:32.445814 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:32.446895 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:32.447449 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:32.448573 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:32.448818 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:32.448912 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:32.451321 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:32.451325 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:32.451328 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:32.451354 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:32.452795 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:32.453274 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:32.453278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:32.453280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:32.453369 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:32.453688 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:32.453692 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:32.453693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:32.453863 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:32.453944 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:32.454412 (+ 468us) server_negotiation.cc:299] Negotiation successful
0927 02:33:32.454532 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 02:33:32.455461 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:32.443819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45855 (local address 127.0.98.1:59839)
0927 02:33:32.444246 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:32.444288 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:32.445497 (+ 1209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:32.445951 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:32.445958 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:32.446291 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:33:32.446759 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:32.446772 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:32.447578 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:32.447581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:32.448457 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:32.448466 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:32.449093 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:32.449096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:32.449238 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:32.449764 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:33:32.449779 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:33:32.451161 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:33:32.452900 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:32.452906 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:32.452919 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:32.453185 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:32.453446 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:32.453449 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:32.453451 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:32.453582 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:32.453955 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:33:32.453961 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:33:32.454178 (+ 217us) client_negotiation.cc:769] Sending connection context
0927 02:33:32.454333 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 02:33:32.454626 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":113,"threads_started":1}
I20260927 02:33:32.456553 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45855
I20260927 02:33:32.456785 635 heartbeater.cc:464] Registering TS with master...
I20260927 02:33:32.457363 635 heartbeater.cc:511] Master 127.0.98.62:45855 requested a full tablet report, sending...
I20260927 02:33:32.458949 444 ts_manager.cc:194] Registered new tserver with Master: cf03610db13a4a3d9533f5858ea439d9 (127.0.98.1:44369)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:33:32.460449 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59839
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:33:32.475711 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45855
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:33:32.587555 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:33:32.587848 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:33:32.587890 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:33:32.592341 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:33:32.592468 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:33:32.592549 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:33:32.599454 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45855
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:33:32.600754 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:33:32.601840 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:33:32.611238 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:32.611436 640 server_base.cc:1034] running on GCE node
W20260927 02:33:32.611280 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:32.611732 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:32.612160 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:33:32.612784 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:33:32.613943 640 hybrid_clock.cc:648] HybridClock initialized: now 1790476412613919 us; error 52 us; skew 500 ppm
Sep 27 02:33:32 dist-test-slave-grlk krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:33:32.620587 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:33:32.621480 640 webserver.cc:460] Webserver started at http://127.0.98.2:38471/ using document root <none> and password file <none>
I20260927 02:33:32.622020 640 fs_manager.cc:359] Metadata directory not provided
I20260927 02:33:32.622083 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:33:32.622259 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:33:32.624217 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/ts-1/data/instance:
uuid: "bf08b3e00afb4ea29a2190859365c0e2"
format_stamp: "Formatted at 2026-09-27 02:33:32 on dist-test-slave-grlk"
server_key: "d17d0afaeee9ff6e29699682f186edfb"
server_key_iv: "8e33fef72d4861e54a44a0f9626ecb60"
server_key_version: "encryptionkey@0"
I20260927 02:33:32.624733 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bf08b3e00afb4ea29a2190859365c0e2"
format_stamp: "Formatted at 2026-09-27 02:33:32 on dist-test-slave-grlk"
server_key: "d17d0afaeee9ff6e29699682f186edfb"
server_key_iv: "8e33fef72d4861e54a44a0f9626ecb60"
server_key_version: "encryptionkey@0"
I20260927 02:33:32.628257 640 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 02:33:32.630650 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:32.631772 640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:33:32.631897 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/ts-1/wal
uuid: "bf08b3e00afb4ea29a2190859365c0e2"
format_stamp: "Formatted at 2026-09-27 02:33:32 on dist-test-slave-grlk"
server_key: "d17d0afaeee9ff6e29699682f186edfb"
server_key_iv: "8e33fef72d4861e54a44a0f9626ecb60"
server_key_version: "encryptionkey@0"
I20260927 02:33:32.631990 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:33:32.641471 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:33:32.644243 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:33:32.644400 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:33:32.644912 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:33:32.645717 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:33:32.645762 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:32.645810 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:33:32.645838 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:32.654752 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35359
I20260927 02:33:32.654778 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35359 every 8 connection(s)
I20260927 02:33:32.655541 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:33:32.662744 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 640
I20260927 02:33:32.662907 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:33:32.663189 392 external_mini_cluster.cc:1428] Setting key fb5720d0c4c3d5440343bca8dbacc7d1
Sep 27 02:33:32 dist-test-slave-grlk krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:33:32.668385 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:32.657189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39259 (local address 127.0.98.62:45855)
0927 02:33:32.657315 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:32.657319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:32.658088 (+ 769us) server_negotiation.cc:407] Connection header received
0927 02:33:32.659369 (+ 1281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:32.659372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:32.659429 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:32.659508 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:32.660573 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:32.661128 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:32.662196 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:32.662407 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:32.662499 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:32.665064 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:32.665068 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:32.665070 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:32.665095 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:32.666380 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:32.666937 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:32.666940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:32.666941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:32.667019 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:32.667338 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:32.667341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:32.667343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:32.667558 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:32.667647 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:32.668111 (+ 464us) server_negotiation.cc:299] Negotiation successful
0927 02:33:32.668226 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 02:33:32.669121 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:32.657550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45855 (local address 127.0.98.2:39259)
0927 02:33:32.657954 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:32.657998 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:32.659173 (+ 1175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:32.659664 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:32.659670 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:32.660002 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:33:32.660437 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:32.660450 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:32.661240 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:32.661242 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:32.662086 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:32.662094 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:32.662553 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:32.662557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:32.662708 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:32.663296 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:33:32.663316 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:33:32.664904 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:33:32.666488 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:32.666493 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:32.666507 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:32.666840 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:32.667098 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:32.667101 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:32.667102 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:32.667249 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:32.667650 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:33:32.667654 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:33:32.667904 (+ 250us) client_negotiation.cc:769] Sending connection context
0927 02:33:32.668049 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 02:33:32.668305 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":105,"threads_started":1}
I20260927 02:33:32.670171 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45855
I20260927 02:33:32.670409 769 heartbeater.cc:464] Registering TS with master...
I20260927 02:33:32.670939 769 heartbeater.cc:511] Master 127.0.98.62:45855 requested a full tablet report, sending...
I20260927 02:33:32.671984 444 ts_manager.cc:194] Registered new tserver with Master: bf08b3e00afb4ea29a2190859365c0e2 (127.0.98.2:35359)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:33:32.672516 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39259
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:33:32.687918 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45855
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:33:32.793802 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:33:32.794094 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:33:32.794165 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:33:32.798257 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:33:32.798321 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:33:32.798416 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:33:32.803560 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45855
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:33:32.804632 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:33:32.805647 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:33:32.814956 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:32.815050 774 server_base.cc:1034] running on GCE node
W20260927 02:33:32.814957 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:32.815238 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:32.815578 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:33:32.816166 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:33:32.817301 774 hybrid_clock.cc:648] HybridClock initialized: now 1790476412817272 us; error 44 us; skew 500 ppm
Sep 27 02:33:32 dist-test-slave-grlk krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:33:32.819701 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:33:32.820473 774 webserver.cc:460] Webserver started at http://127.0.98.3:46805/ using document root <none> and password file <none>
I20260927 02:33:32.820973 774 fs_manager.cc:359] Metadata directory not provided
I20260927 02:33:32.821041 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:33:32.821195 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:33:32.823072 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/ts-2/data/instance:
uuid: "e2357903bebb4ebe95059996bc4b0c3a"
format_stamp: "Formatted at 2026-09-27 02:33:32 on dist-test-slave-grlk"
server_key: "e3163eb8a655e723bee8c1526d07d36e"
server_key_iv: "8ff1b036835d897c04ef40bd446faed4"
server_key_version: "encryptionkey@0"
I20260927 02:33:32.823556 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e2357903bebb4ebe95059996bc4b0c3a"
format_stamp: "Formatted at 2026-09-27 02:33:32 on dist-test-slave-grlk"
server_key: "e3163eb8a655e723bee8c1526d07d36e"
server_key_iv: "8ff1b036835d897c04ef40bd446faed4"
server_key_version: "encryptionkey@0"
I20260927 02:33:32.826921 774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:33:32.829159 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:32.830334 774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:33:32.830432 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/ts-2/wal
uuid: "e2357903bebb4ebe95059996bc4b0c3a"
format_stamp: "Formatted at 2026-09-27 02:33:32 on dist-test-slave-grlk"
server_key: "e3163eb8a655e723bee8c1526d07d36e"
server_key_iv: "8ff1b036835d897c04ef40bd446faed4"
server_key_version: "encryptionkey@0"
I20260927 02:33:32.830521 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:33:32.845801 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:33:32.848354 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:33:32.848516 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:33:32.849076 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:33:32.849818 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:33:32.849861 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:32.849910 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:33:32.849936 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:32.859710 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38761
I20260927 02:33:32.859750 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38761 every 8 connection(s)
I20260927 02:33:32.860569 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:33:32.862980 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 774
I20260927 02:33:32.863072 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476410933508-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:33:32.863324 392 external_mini_cluster.cc:1428] Setting key c93c14928c7fcd0994c2eb78472df944
Sep 27 02:33:32 dist-test-slave-grlk krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:33:32.872803 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:32.862280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57765 (local address 127.0.98.62:45855)
0927 02:33:32.862420 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:32.862424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:32.863256 (+ 832us) server_negotiation.cc:407] Connection header received
0927 02:33:32.864561 (+ 1305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:32.864564 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:32.864617 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:32.864691 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:32.865845 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:32.866356 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:32.867429 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:32.867638 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:32.867722 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:32.869917 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:32.869920 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:32.869922 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:32.869944 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:32.871013 (+ 1069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:32.871490 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:32.871493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:32.871494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:32.871563 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:32.871846 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:32.871849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:32.871851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:32.872021 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:32.872102 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:32.872542 (+ 440us) server_negotiation.cc:299] Negotiation successful
0927 02:33:32.872650 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 02:33:32.873618 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:32.862637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45855 (local address 127.0.98.3:57765)
0927 02:33:32.863094 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:32.863145 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:32.864366 (+ 1221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:32.864871 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:32.864878 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:32.865252 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:33:32.865715 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:32.865728 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:32.866462 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:32.866466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:32.867322 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:32.867330 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:32.867761 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:32.867765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:32.867920 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:32.868423 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:33:32.868438 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:33:32.869777 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:33:32.871111 (+ 1334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:32.871116 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:32.871129 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:32.871400 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:32.871640 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:32.871643 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:32.871645 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:32.871765 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:32.872112 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:33:32.872118 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:33:32.872333 (+ 215us) client_negotiation.cc:769] Sending connection context
0927 02:33:32.872487 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 02:33:32.872784 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":111,"threads_started":1}
I20260927 02:33:32.874708 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45855
I20260927 02:33:32.874951 903 heartbeater.cc:464] Registering TS with master...
I20260927 02:33:32.875396 903 heartbeater.cc:511] Master 127.0.98.62:45855 requested a full tablet report, sending...
I20260927 02:33:32.876348 443 ts_manager.cc:194] Registered new tserver with Master: e2357903bebb4ebe95059996bc4b0c3a (127.0.98.3:38761)
I20260927 02:33:32.876992 443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57765
I20260927 02:33:32.877112 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 02:33:32 dist-test-slave-grlk krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476412, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 02:33:32 dist-test-slave-grlk krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476412, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:33:32.901049 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:32.892070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49920 (local address 127.0.98.1:44369)
0927 02:33:32.892904 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:32.892913 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:32.892972 (+ 59us) server_negotiation.cc:407] Connection header received
0927 02:33:32.893034 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:32.893039 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:32.893226 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:32.893346 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:32.893981 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:32.894760 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:32.895630 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:32.896030 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:32.896105 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:32.898302 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:32.898309 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:32.898321 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:32.898342 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:32.899634 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:32.899980 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:32.899984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:32.899986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:32.900054 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:32.900231 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:32.900233 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:32.900234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:32.900443 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:32.900588 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:32.900719 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 02:33:32.900827 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":715,"thread_start_us":616,"threads_started":1}
W20260927 02:33:32.901916 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49920
Sep 27 02:33:32 dist-test-slave-grlk krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 02:33:32 dist-test-slave-grlk krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:33:32.924803 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:32.917462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49936 (local address 127.0.98.1:44369)
0927 02:33:32.917588 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:32.917592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:32.917715 (+ 123us) server_negotiation.cc:407] Connection header received
0927 02:33:32.917829 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:32.917831 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:32.917881 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:32.917955 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:32.918583 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:32.919124 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:32.920028 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:32.920258 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:32.920352 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:32.922546 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:32.922549 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:32.922551 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:32.922572 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:32.923643 (+ 1071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:32.923967 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:32.923969 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:32.923971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:32.924040 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:32.924218 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:32.924220 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:32.924221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:32.924355 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:32.924438 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:32.924603 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 02:33:32.924687 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 02:33:32.925813 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 506
I20260927 02:33:32.931514 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 640
I20260927 02:33:32.938000 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 774
I20260927 02:33:32.942780 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 412
2026-09-27T02:33:32Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2002 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:33:32 dist-test-slave-grlk krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:33:32 dist-test-slave-grlk krb5kdc[917](info): set up 2 sockets
Sep 27 02:33:32 dist-test-slave-grlk krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260927 02:33:33.900382 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/krb5kdc/test-user.keytab.
Sep 27 02:33:33 dist-test-slave-grlk krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:33:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:33:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:33:34.009563 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35845
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35845
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:33:34.113579 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:33:34.113864 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:33:34.113903 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:33:34.117835 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:33:34.117888 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:33:34.117906 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:33:34.117921 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:33:34.117936 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:33:34.123459 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:33:34.124544 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:33:34.125612 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:33:34.133744 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:34.133762 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:34.133765 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:34.133844 930 server_base.cc:1034] running on GCE node
I20260927 02:33:34.134456 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:33:34.135492 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:33:34.136647 930 hybrid_clock.cc:648] HybridClock initialized: now 1790476414136632 us; error 28 us; skew 500 ppm
Sep 27 02:33:34 dist-test-slave-grlk krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476414, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:33:34.139196 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:33:34.140014 930 webserver.cc:460] Webserver started at http://127.0.98.62:46519/ using document root <none> and password file <none>
I20260927 02:33:34.140523 930 fs_manager.cc:359] Metadata directory not provided
I20260927 02:33:34.140585 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:33:34.140774 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:33:34.142731 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "4df68f26a84840b09a3581138056fda7"
format_stamp: "Formatted at 2026-09-27 02:33:34 on dist-test-slave-grlk"
server_key: "d78f045f4ea113a4e6d7f00f8be738b5"
server_key_iv: "86d1eb35337dda1e2cca29f52535b5c1"
server_key_version: "encryptionkey@0"
I20260927 02:33:34.143235 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "4df68f26a84840b09a3581138056fda7"
format_stamp: "Formatted at 2026-09-27 02:33:34 on dist-test-slave-grlk"
server_key: "d78f045f4ea113a4e6d7f00f8be738b5"
server_key_iv: "86d1eb35337dda1e2cca29f52535b5c1"
server_key_version: "encryptionkey@0"
I20260927 02:33:34.146622 930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:33:34.148758 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:34.149899 930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 02:33:34.150002 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "4df68f26a84840b09a3581138056fda7"
format_stamp: "Formatted at 2026-09-27 02:33:34 on dist-test-slave-grlk"
server_key: "d78f045f4ea113a4e6d7f00f8be738b5"
server_key_iv: "86d1eb35337dda1e2cca29f52535b5c1"
server_key_version: "encryptionkey@0"
I20260927 02:33:34.150089 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:33:34.167886 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:33:34.170917 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:33:34.171083 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:33:34.177506 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35845
I20260927 02:33:34.177516 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35845 every 8 connection(s)
I20260927 02:33:34.178381 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:33:34.181016 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:33:34.184842 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 930
I20260927 02:33:34.184965 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:33:34.185187 392 external_mini_cluster.cc:1428] Setting key fda52e75648b398eccfdda25a1cd129f
I20260927 02:33:34.186514 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4df68f26a84840b09a3581138056fda7: Bootstrap starting.
I20260927 02:33:34.188763 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4df68f26a84840b09a3581138056fda7: Neither blocks nor log segments found. Creating new log.
I20260927 02:33:34.189636 998 log.cc:824] T 00000000000000000000000000000000 P 4df68f26a84840b09a3581138056fda7: Log is configured to *not* fsync() on all Append() calls
I20260927 02:33:34.191479 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4df68f26a84840b09a3581138056fda7: No bootstrap required, opened a new log
Sep 27 02:33:34 dist-test-slave-grlk krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:33:34.195302 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4df68f26a84840b09a3581138056fda7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df68f26a84840b09a3581138056fda7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35845 } }
I20260927 02:33:34.195573 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4df68f26a84840b09a3581138056fda7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:33:34.195648 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4df68f26a84840b09a3581138056fda7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4df68f26a84840b09a3581138056fda7, State: Initialized, Role: FOLLOWER
I20260927 02:33:34.196128 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4df68f26a84840b09a3581138056fda7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df68f26a84840b09a3581138056fda7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35845 } }
I20260927 02:33:34.196278 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4df68f26a84840b09a3581138056fda7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:33:34.196346 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4df68f26a84840b09a3581138056fda7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:33:34.196435 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4df68f26a84840b09a3581138056fda7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:33:34.197477 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4df68f26a84840b09a3581138056fda7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df68f26a84840b09a3581138056fda7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35845 } }
I20260927 02:33:34.197829 998 leader_election.cc:302] T 00000000000000000000000000000000 P 4df68f26a84840b09a3581138056fda7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4df68f26a84840b09a3581138056fda7; no voters:
I20260927 02:33:34.198153 998 leader_election.cc:288] T 00000000000000000000000000000000 P 4df68f26a84840b09a3581138056fda7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:33:34.198518 1004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4df68f26a84840b09a3581138056fda7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:33:34.198719 1004 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4df68f26a84840b09a3581138056fda7 [term 1 LEADER]: Becoming Leader. State: Replica: 4df68f26a84840b09a3581138056fda7, State: Running, Role: LEADER
I20260927 02:33:34.199090 1004 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4df68f26a84840b09a3581138056fda7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df68f26a84840b09a3581138056fda7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35845 } }
I20260927 02:33:34.199257 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:34.186505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43666 (local address 127.0.98.62:35845)
0927 02:33:34.186888 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:34.186904 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:34.186936 (+ 32us) server_negotiation.cc:407] Connection header received
0927 02:33:34.187537 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:34.187553 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:34.187860 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:34.188133 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:34.188994 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:34.190312 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:34.191143 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:34.191811 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:34.191964 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:34.194326 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:34.194332 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:34.194348 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:34.194378 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:34.196204 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:34.196696 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:34.196702 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:34.196707 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:34.196836 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:34.197108 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:34.197111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:34.197113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:34.197636 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:34.197816 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:34.198012 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 02:33:34.198304 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":111,"threads_started":1}
I20260927 02:33:34.199409 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4df68f26a84840b09a3581138056fda7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:33:34.201635 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4df68f26a84840b09a3581138056fda7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4df68f26a84840b09a3581138056fda7. Latest consensus state: current_term: 1 leader_uuid: "4df68f26a84840b09a3581138056fda7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df68f26a84840b09a3581138056fda7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35845 } } }
I20260927 02:33:34.201804 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4df68f26a84840b09a3581138056fda7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4df68f26a84840b09a3581138056fda7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df68f26a84840b09a3581138056fda7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35845 } } }
I20260927 02:33:34.202258 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4df68f26a84840b09a3581138056fda7 [sys.catalog]: This master's current role is: LEADER
I20260927 02:33:34.202266 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4df68f26a84840b09a3581138056fda7 [sys.catalog]: This master's current role is: LEADER
W20260927 02:33:34.204921 1017 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4df68f26a84840b09a3581138056fda7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:33:34.205030 1017 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:33:34.205132 1018 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:33:34.205884 1018 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:33:34.210906 1018 catalog_manager.cc:1347] Generated new cluster ID: efb99f5b4b434d12a60b00af33c07344
I20260927 02:33:34.210966 1018 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:33:34.228792 1018 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:33:34.229712 1018 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:33:34.236402 1018 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4df68f26a84840b09a3581138056fda7: Generated new TSK 0
I20260927 02:33:34.237078 1018 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:33:34.266055 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35845
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:33:34.386154 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:33:34.386370 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:33:34.386410 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:33:34.390215 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:33:34.390259 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:33:34.390312 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:33:34.395201 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35845
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:33:34.396095 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:33:34.397015 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:33:34.405571 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:34.405579 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:34.405665 1024 server_base.cc:1034] running on GCE node
W20260927 02:33:34.405586 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:34.406191 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:33:34.406739 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:33:34.407889 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790476414407868 us; error 37 us; skew 500 ppm
Sep 27 02:33:34 dist-test-slave-grlk krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476414, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:33:34.410641 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:33:34.411381 1024 webserver.cc:460] Webserver started at http://127.0.98.1:43837/ using document root <none> and password file <none>
I20260927 02:33:34.411832 1024 fs_manager.cc:359] Metadata directory not provided
I20260927 02:33:34.411883 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:33:34.412040 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:33:34.413875 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/ts-0/data/instance:
uuid: "8e36a9146c5648709151cce5d91869c2"
format_stamp: "Formatted at 2026-09-27 02:33:34 on dist-test-slave-grlk"
server_key: "e5ccd0e92119a88c521d2cd29fbd0e49"
server_key_iv: "b36c9823e7ebbf5fdab7b10822e2c38c"
server_key_version: "encryptionkey@0"
I20260927 02:33:34.414338 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8e36a9146c5648709151cce5d91869c2"
format_stamp: "Formatted at 2026-09-27 02:33:34 on dist-test-slave-grlk"
server_key: "e5ccd0e92119a88c521d2cd29fbd0e49"
server_key_iv: "b36c9823e7ebbf5fdab7b10822e2c38c"
server_key_version: "encryptionkey@0"
I20260927 02:33:34.417493 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:33:34.419526 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:34.420545 1024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 02:33:34.420629 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/ts-0/wal
uuid: "8e36a9146c5648709151cce5d91869c2"
format_stamp: "Formatted at 2026-09-27 02:33:34 on dist-test-slave-grlk"
server_key: "e5ccd0e92119a88c521d2cd29fbd0e49"
server_key_iv: "b36c9823e7ebbf5fdab7b10822e2c38c"
server_key_version: "encryptionkey@0"
I20260927 02:33:34.420702 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:33:34.441310 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:33:34.443681 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:33:34.443811 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:33:34.444280 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:33:34.444995 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:33:34.445039 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:34.445073 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:33:34.445093 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:34.452893 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36507
I20260927 02:33:34.452915 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36507 every 8 connection(s)
I20260927 02:33:34.453809 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:33:34.461809 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 1024
I20260927 02:33:34.461889 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:33:34.462116 392 external_mini_cluster.cc:1428] Setting key cfe6fac30b3382a6783706f8b5972463
Sep 27 02:33:34 dist-test-slave-grlk krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476414, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:33:34.466320 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:34.455390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48675 (local address 127.0.98.62:35845)
0927 02:33:34.455505 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:34.455509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:34.456253 (+ 744us) server_negotiation.cc:407] Connection header received
0927 02:33:34.457568 (+ 1315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:34.457571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:34.457628 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:34.457700 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:34.458789 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:34.459321 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:34.460396 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:34.460624 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:34.460709 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:34.462968 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:34.462972 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:34.462974 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:34.462997 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:34.464304 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:34.464837 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:34.464840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:34.464841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:34.464918 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:34.465286 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:34.465288 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:34.465290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:34.465447 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:34.465518 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:34.466048 (+ 530us) server_negotiation.cc:299] Negotiation successful
0927 02:33:34.466167 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 02:33:34.466976 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:34.455716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35845 (local address 127.0.98.1:48675)
0927 02:33:34.456115 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:34.456154 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:34.457360 (+ 1206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:34.457867 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:34.457874 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:34.458208 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:33:34.458652 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:34.458664 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:34.459431 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:34.459434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:34.460280 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:34.460289 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:34.460749 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:34.460753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:34.460903 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:34.461421 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:33:34.461435 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:33:34.462828 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:33:34.464408 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:34.464413 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:34.464426 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:34.464743 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:34.465041 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:34.465043 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:34.465045 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:34.465206 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:34.465534 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:33:34.465541 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:33:34.465814 (+ 273us) client_negotiation.cc:769] Sending connection context
0927 02:33:34.465966 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 02:33:34.466204 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":91,"threads_started":1}
I20260927 02:33:34.467984 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35845
I20260927 02:33:34.468195 1153 heartbeater.cc:464] Registering TS with master...
I20260927 02:33:34.468617 1153 heartbeater.cc:511] Master 127.0.98.62:35845 requested a full tablet report, sending...
I20260927 02:33:34.469985 962 ts_manager.cc:194] Registered new tserver with Master: 8e36a9146c5648709151cce5d91869c2 (127.0.98.1:36507)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:33:34.471055 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48675
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:33:34.485334 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35845
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:33:34.589985 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:33:34.590204 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:33:34.590245 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:33:34.594115 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:33:34.594157 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:33:34.594210 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:33:34.599108 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35845
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:33:34.600014 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:33:34.600987 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:33:34.609714 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:34.609758 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:34.609721 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:34.610034 1158 server_base.cc:1034] running on GCE node
I20260927 02:33:34.610392 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:33:34.610984 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:33:34.612170 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790476414612136 us; error 44 us; skew 500 ppm
Sep 27 02:33:34 dist-test-slave-grlk krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476414, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:33:34.615020 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:33:34.615852 1158 webserver.cc:460] Webserver started at http://127.0.98.2:43753/ using document root <none> and password file <none>
I20260927 02:33:34.616369 1158 fs_manager.cc:359] Metadata directory not provided
I20260927 02:33:34.616441 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:33:34.616641 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:33:34.618695 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/ts-1/data/instance:
uuid: "09da89815e9048cda1338c034c6cb579"
format_stamp: "Formatted at 2026-09-27 02:33:34 on dist-test-slave-grlk"
server_key: "9c8bdb5084e501c99854b63282ae1713"
server_key_iv: "8159855648bb812f31a80c690fc3fd72"
server_key_version: "encryptionkey@0"
I20260927 02:33:34.619194 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/ts-1/wal/instance:
uuid: "09da89815e9048cda1338c034c6cb579"
format_stamp: "Formatted at 2026-09-27 02:33:34 on dist-test-slave-grlk"
server_key: "9c8bdb5084e501c99854b63282ae1713"
server_key_iv: "8159855648bb812f31a80c690fc3fd72"
server_key_version: "encryptionkey@0"
I20260927 02:33:34.622784 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:33:34.625236 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:34.626408 1158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 02:33:34.626511 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/ts-1/wal
uuid: "09da89815e9048cda1338c034c6cb579"
format_stamp: "Formatted at 2026-09-27 02:33:34 on dist-test-slave-grlk"
server_key: "9c8bdb5084e501c99854b63282ae1713"
server_key_iv: "8159855648bb812f31a80c690fc3fd72"
server_key_version: "encryptionkey@0"
I20260927 02:33:34.626613 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:33:34.642824 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:33:34.645385 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:33:34.645553 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:33:34.646080 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:33:34.646850 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:33:34.646895 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:34.646946 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:33:34.646972 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:34.655277 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35647
I20260927 02:33:34.655321 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35647 every 8 connection(s)
I20260927 02:33:34.656232 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:33:34.660656 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 1158
I20260927 02:33:34.660777 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:33:34.661073 392 external_mini_cluster.cc:1428] Setting key b6a1f17aaecf2be3b27e9c18a8843d39
Sep 27 02:33:34 dist-test-slave-grlk krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476414, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:33:34.671526 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:34.658156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56315 (local address 127.0.98.62:35845)
0927 02:33:34.658309 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:34.658314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:34.659157 (+ 843us) server_negotiation.cc:407] Connection header received
0927 02:33:34.660425 (+ 1268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:34.660428 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:34.660475 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:34.660552 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:34.662008 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:34.662619 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:34.664199 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:34.664542 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:34.664652 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:34.667280 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:34.667283 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:34.667285 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:34.667308 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:34.668520 (+ 1212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:34.669185 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:34.669188 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:34.669189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:34.669259 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:34.670579 (+ 1320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:34.670582 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:34.670583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:34.670755 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:34.670828 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:34.671316 (+ 488us) server_negotiation.cc:299] Negotiation successful
0927 02:33:34.671408 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 02:33:34.672359 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:34.658584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35845 (local address 127.0.98.2:56315)
0927 02:33:34.659013 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:34.659065 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:34.660228 (+ 1163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:34.660756 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:34.660766 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:34.661273 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:33:34.661836 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:34.661855 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:34.662746 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:34.662750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:34.664068 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:34.664080 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:34.664680 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:34.664683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:34.664818 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:34.665388 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:33:34.665404 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:33:34.667114 (+ 1710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:33:34.668644 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:34.668651 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:34.668666 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:34.669082 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:34.669343 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:34.669347 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:34.669349 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:34.670467 (+ 1118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:34.670873 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:33:34.670879 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:33:34.671108 (+ 229us) client_negotiation.cc:769] Sending connection context
0927 02:33:34.671275 (+ 167us) client_negotiation.cc:240] Negotiation successful
0927 02:33:34.671544 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":98,"threads_started":1}
I20260927 02:33:34.673440 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35845
I20260927 02:33:34.673651 1287 heartbeater.cc:464] Registering TS with master...
I20260927 02:33:34.674082 1287 heartbeater.cc:511] Master 127.0.98.62:35845 requested a full tablet report, sending...
I20260927 02:33:34.674988 962 ts_manager.cc:194] Registered new tserver with Master: 09da89815e9048cda1338c034c6cb579 (127.0.98.2:35647)
I20260927 02:33:34.675508 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56315
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 02:33:35.473132 1153 heartbeater.cc:503] Master 127.0.98.62:35845 was elected leader, sending a full tablet report...
I20260927 02:33:35.677346 1287 heartbeater.cc:503] Master 127.0.98.62:35845 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260927 02:33:35.800500 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 1.138s user 0.000s sys 0.003s
I20260927 02:33:35.801009 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35845
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:33:35.911396 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:33:35.911633 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:33:35.911675 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:33:35.915553 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:33:35.915598 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:33:35.915664 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:33:35.921020 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35845
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:33:35.922022 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:33:35.923019 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:33:35.931985 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:35.931978 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:35.931974 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:35.932147 1292 server_base.cc:1034] running on GCE node
I20260927 02:33:35.932633 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:33:35.933295 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:33:35.934448 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790476415934422 us; error 42 us; skew 500 ppm
Sep 27 02:33:35 dist-test-slave-grlk krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:33:35.937090 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:33:35.937860 1292 webserver.cc:460] Webserver started at http://127.0.98.3:45921/ using document root <none> and password file <none>
I20260927 02:33:35.938365 1292 fs_manager.cc:359] Metadata directory not provided
I20260927 02:33:35.938431 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:33:35.938593 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:33:35.940552 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/ts-2/data/instance:
uuid: "318814c64d7a4366a2f6a186a1298da9"
format_stamp: "Formatted at 2026-09-27 02:33:35 on dist-test-slave-grlk"
server_key: "88249dfe65066471dff9ab51225b35ee"
server_key_iv: "539cb0f4c9bde39f7be5a8e9c8754c89"
server_key_version: "encryptionkey@0"
I20260927 02:33:35.941097 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/ts-2/wal/instance:
uuid: "318814c64d7a4366a2f6a186a1298da9"
format_stamp: "Formatted at 2026-09-27 02:33:35 on dist-test-slave-grlk"
server_key: "88249dfe65066471dff9ab51225b35ee"
server_key_iv: "539cb0f4c9bde39f7be5a8e9c8754c89"
server_key_version: "encryptionkey@0"
I20260927 02:33:35.944617 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:33:35.946878 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:35.948071 1292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:33:35.948170 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/ts-2/wal
uuid: "318814c64d7a4366a2f6a186a1298da9"
format_stamp: "Formatted at 2026-09-27 02:33:35 on dist-test-slave-grlk"
server_key: "88249dfe65066471dff9ab51225b35ee"
server_key_iv: "539cb0f4c9bde39f7be5a8e9c8754c89"
server_key_version: "encryptionkey@0"
I20260927 02:33:35.948251 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:33:35.975456 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:33:35.978317 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:33:35.978490 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:33:35.979038 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:33:35.979835 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:33:35.979883 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:35.979934 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:33:35.979964 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:35.988340 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39939
I20260927 02:33:35.988377 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39939 every 8 connection(s)
I20260927 02:33:35.989220 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:33:35.996598 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 1292
I20260927 02:33:35.996712 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476410933508-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:33:35.997037 392 external_mini_cluster.cc:1428] Setting key a20eb7d44f2c4e5bf5d3817b08711fc4
Sep 27 02:33:35 dist-test-slave-grlk krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:33:36.002655 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:35.990880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47019 (local address 127.0.98.62:35845)
0927 02:33:35.991134 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:35.991138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:35.991808 (+ 670us) server_negotiation.cc:407] Connection header received
0927 02:33:35.993100 (+ 1292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:35.993103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:35.993159 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:35.993237 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:35.994321 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:35.994847 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:35.995904 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:35.996123 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:35.996209 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:35.998744 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:35.998747 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:35.998749 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:35.998770 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:36.000076 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:36.001322 (+ 1246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:36.001325 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:36.001326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:36.001409 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:36.001713 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:36.001715 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:36.001716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:36.001886 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:36.001959 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:36.002418 (+ 459us) server_negotiation.cc:299] Negotiation successful
0927 02:33:36.002528 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":78,"threads_started":1}
I20260927 02:33:36.003479 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:35.991236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35845 (local address 127.0.98.3:47019)
0927 02:33:35.991669 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:35.991708 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:35.992870 (+ 1162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:35.993406 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:35.993413 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:35.993747 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:33:35.994190 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:35.994203 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:35.994957 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:35.994960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:35.995794 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:35.995802 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:35.996249 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:35.996253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:35.996426 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:35.997025 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:33:35.997043 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:33:35.998609 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:33:36.000174 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:36.000179 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:36.000974 (+ 795us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:36.001221 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:36.001515 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:36.001517 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:36.001519 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:36.001639 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:36.001989 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:33:36.001995 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:33:36.002217 (+ 222us) client_negotiation.cc:769] Sending connection context
0927 02:33:36.002373 (+ 156us) client_negotiation.cc:240] Negotiation successful
0927 02:33:36.002648 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":113,"threads_started":1}
I20260927 02:33:36.004612 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35845
I20260927 02:33:36.004818 1421 heartbeater.cc:464] Registering TS with master...
I20260927 02:33:36.005391 1421 heartbeater.cc:511] Master 127.0.98.62:35845 requested a full tablet report, sending...
I20260927 02:33:36.006393 962 ts_manager.cc:194] Registered new tserver with Master: 318814c64d7a4366a2f6a186a1298da9 (127.0.98.3:39939)
I20260927 02:33:36.007009 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47019
I20260927 02:33:36.010732 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:33:36.021195 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:36.014068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43682 (local address 127.0.98.62:35845)
0927 02:33:36.014200 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:36.014204 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:36.014328 (+ 124us) server_negotiation.cc:407] Connection header received
0927 02:33:36.014454 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:36.014457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:36.014499 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:36.014581 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:36.015306 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:36.015817 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:36.016667 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:36.016896 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:36.017040 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:36.018329 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:36.018333 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:36.018336 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:36.018373 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:36.019479 (+ 1106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:36.019886 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:36.019890 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:36.019892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:36.019973 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:36.020236 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:36.020240 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:36.020242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:36.020464 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:36.020623 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:36.020870 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 02:33:36.021023 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 02:33:36.025696 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43682:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:33:36.028657 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:33:36.042925 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:36.038020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39939 (local address 127.0.0.1:58544)
0927 02:33:36.038621 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:36.038633 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:36.038714 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:36.039255 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:36.039257 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:36.039273 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:33:36.039348 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:36.039354 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:36.040280 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:36.040284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:36.041644 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:36.041655 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:36.042428 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:36.042432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:36.042530 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:36.042565 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 02:33:36.042663 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 02:33:36.042779 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":1,"thread_start_us":76,"threads_started":1}
I20260927 02:33:36.042930 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:36.038276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58544 (local address 127.0.98.3:39939)
0927 02:33:36.038869 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:36.038880 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:36.038899 (+ 19us) server_negotiation.cc:407] Connection header received
0927 02:33:36.038961 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:36.038966 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:36.039138 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:36.039271 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:33:36.039452 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:36.040168 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:36.041780 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:36.042323 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:36.042417 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:36.042556 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:36.042696 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 02:33:36.042780 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":165,"threads_started":1}
I20260927 02:33:36.043267 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:36.037254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36507 (local address 127.0.0.1:57286)
0927 02:33:36.038082 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:36.038164 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:36.038300 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:36.039017 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:36.039019 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:36.039033 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:33:36.039107 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:36.039111 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:36.040524 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:36.040527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:36.041604 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:36.041613 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:36.042813 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:36.042817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:36.042948 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:36.043006 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 02:33:36.043077 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 02:33:36.043143 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"mutex_wait_us":74,"thread_start_us":131,"threads_started":1}
I20260927 02:33:36.043586 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:36.037292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35647 (local address 127.0.0.1:59446)
0927 02:33:36.038291 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:36.038303 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:36.038373 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:36.038911 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:36.038917 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:36.038939 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:33:36.039063 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:36.039069 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:36.040488 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:36.040491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:36.041679 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:36.041686 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:36.042813 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:36.042817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:36.042951 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:36.043324 (+ 373us) client_negotiation.cc:769] Sending connection context
0927 02:33:36.043392 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 02:33:36.043459 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"mutex_wait_us":50,"thread_start_us":86,"threads_started":1}
I20260927 02:33:36.043637 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:36.037361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57286 (local address 127.0.98.1:36507)
0927 02:33:36.037877 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:36.037885 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:36.038439 (+ 554us) server_negotiation.cc:407] Connection header received
0927 02:33:36.038525 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:36.038530 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:36.038729 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:36.038868 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:33:36.039223 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:36.040370 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:36.041712 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:36.042525 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:36.042628 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:36.042750 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:36.043420 (+ 670us) server_negotiation.cc:299] Negotiation successful
0927 02:33:36.043515 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":105,"threads_started":1}
I20260927 02:33:36.044592 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:36.037512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59446 (local address 127.0.98.2:35647)
0927 02:33:36.038040 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:36.038046 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:36.038449 (+ 403us) server_negotiation.cc:407] Connection header received
0927 02:33:36.038525 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:36.038530 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:36.038729 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:36.038868 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:33:36.039182 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:36.040370 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:36.041764 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:36.042528 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:36.042628 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:36.042750 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:36.044351 (+ 1601us) server_negotiation.cc:299] Negotiation successful
0927 02:33:36.044445 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":87,"threads_started":1}
I20260927 02:33:36.046111 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 0d40e889b77e4a5ba5e4cddf3ae55b40 (DEFAULT_TABLE table=test-table [id=1f897709c13b4ed58e4fb4d4afbe2144]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:33:36.046337 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 0d40e889b77e4a5ba5e4cddf3ae55b40 (DEFAULT_TABLE table=test-table [id=1f897709c13b4ed58e4fb4d4afbe2144]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:33:36.047295 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d40e889b77e4a5ba5e4cddf3ae55b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:33:36.047302 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d40e889b77e4a5ba5e4cddf3ae55b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:33:36.050117 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 0d40e889b77e4a5ba5e4cddf3ae55b40 (DEFAULT_TABLE table=test-table [id=1f897709c13b4ed58e4fb4d4afbe2144]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:33:36.051318 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d40e889b77e4a5ba5e4cddf3ae55b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:33:36.055469 1440 tablet_bootstrap.cc:492] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 8e36a9146c5648709151cce5d91869c2: Bootstrap starting.
I20260927 02:33:36.056239 1441 tablet_bootstrap.cc:492] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 318814c64d7a4366a2f6a186a1298da9: Bootstrap starting.
I20260927 02:33:36.057889 1440 tablet_bootstrap.cc:654] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 8e36a9146c5648709151cce5d91869c2: Neither blocks nor log segments found. Creating new log.
I20260927 02:33:36.057927 1441 tablet_bootstrap.cc:654] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 318814c64d7a4366a2f6a186a1298da9: Neither blocks nor log segments found. Creating new log.
I20260927 02:33:36.058857 1442 tablet_bootstrap.cc:492] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 09da89815e9048cda1338c034c6cb579: Bootstrap starting.
I20260927 02:33:36.059073 1441 log.cc:824] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 318814c64d7a4366a2f6a186a1298da9: Log is configured to *not* fsync() on all Append() calls
I20260927 02:33:36.059173 1440 log.cc:824] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 8e36a9146c5648709151cce5d91869c2: Log is configured to *not* fsync() on all Append() calls
I20260927 02:33:36.061134 1441 tablet_bootstrap.cc:492] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 318814c64d7a4366a2f6a186a1298da9: No bootstrap required, opened a new log
I20260927 02:33:36.061344 1441 ts_tablet_manager.cc:1397] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 318814c64d7a4366a2f6a186a1298da9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 02:33:36.061455 1442 tablet_bootstrap.cc:654] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 09da89815e9048cda1338c034c6cb579: Neither blocks nor log segments found. Creating new log.
I20260927 02:33:36.062196 1442 log.cc:824] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 09da89815e9048cda1338c034c6cb579: Log is configured to *not* fsync() on all Append() calls
I20260927 02:33:36.062812 1440 tablet_bootstrap.cc:492] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 8e36a9146c5648709151cce5d91869c2: No bootstrap required, opened a new log
I20260927 02:33:36.062995 1440 ts_tablet_manager.cc:1397] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 8e36a9146c5648709151cce5d91869c2: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260927 02:33:36.064075 1442 tablet_bootstrap.cc:492] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 09da89815e9048cda1338c034c6cb579: No bootstrap required, opened a new log
I20260927 02:33:36.064273 1442 ts_tablet_manager.cc:1397] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 09da89815e9048cda1338c034c6cb579: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 02:33:36.064406 1441 raft_consensus.cc:348] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 318814c64d7a4366a2f6a186a1298da9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e36a9146c5648709151cce5d91869c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36507 } } peers { permanent_uuid: "09da89815e9048cda1338c034c6cb579" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35647 } } peers { permanent_uuid: "318814c64d7a4366a2f6a186a1298da9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39939 } }
I20260927 02:33:36.064630 1441 raft_consensus.cc:374] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 318814c64d7a4366a2f6a186a1298da9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:33:36.064687 1441 raft_consensus.cc:729] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 318814c64d7a4366a2f6a186a1298da9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 318814c64d7a4366a2f6a186a1298da9, State: Initialized, Role: FOLLOWER
I20260927 02:33:36.065164 1441 consensus_queue.cc:260] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 318814c64d7a4366a2f6a186a1298da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e36a9146c5648709151cce5d91869c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36507 } } peers { permanent_uuid: "09da89815e9048cda1338c034c6cb579" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35647 } } peers { permanent_uuid: "318814c64d7a4366a2f6a186a1298da9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39939 } }
I20260927 02:33:36.065748 1421 heartbeater.cc:503] Master 127.0.98.62:35845 was elected leader, sending a full tablet report...
I20260927 02:33:36.066093 1441 ts_tablet_manager.cc:1428] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 318814c64d7a4366a2f6a186a1298da9: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 02:33:36.067268 1440 raft_consensus.cc:348] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 8e36a9146c5648709151cce5d91869c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e36a9146c5648709151cce5d91869c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36507 } } peers { permanent_uuid: "09da89815e9048cda1338c034c6cb579" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35647 } } peers { permanent_uuid: "318814c64d7a4366a2f6a186a1298da9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39939 } }
I20260927 02:33:36.067529 1440 raft_consensus.cc:374] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 8e36a9146c5648709151cce5d91869c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:33:36.067585 1440 raft_consensus.cc:729] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 8e36a9146c5648709151cce5d91869c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e36a9146c5648709151cce5d91869c2, State: Initialized, Role: FOLLOWER
I20260927 02:33:36.068017 1440 consensus_queue.cc:260] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 8e36a9146c5648709151cce5d91869c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e36a9146c5648709151cce5d91869c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36507 } } peers { permanent_uuid: "09da89815e9048cda1338c034c6cb579" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35647 } } peers { permanent_uuid: "318814c64d7a4366a2f6a186a1298da9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39939 } }
I20260927 02:33:36.068228 1442 raft_consensus.cc:348] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 09da89815e9048cda1338c034c6cb579 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e36a9146c5648709151cce5d91869c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36507 } } peers { permanent_uuid: "09da89815e9048cda1338c034c6cb579" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35647 } } peers { permanent_uuid: "318814c64d7a4366a2f6a186a1298da9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39939 } }
I20260927 02:33:36.068464 1442 raft_consensus.cc:374] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 09da89815e9048cda1338c034c6cb579 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:33:36.068531 1442 raft_consensus.cc:729] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 09da89815e9048cda1338c034c6cb579 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09da89815e9048cda1338c034c6cb579, State: Initialized, Role: FOLLOWER
I20260927 02:33:36.068990 1440 ts_tablet_manager.cc:1428] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 8e36a9146c5648709151cce5d91869c2: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 02:33:36.069020 1442 consensus_queue.cc:260] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 09da89815e9048cda1338c034c6cb579 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e36a9146c5648709151cce5d91869c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36507 } } peers { permanent_uuid: "09da89815e9048cda1338c034c6cb579" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35647 } } peers { permanent_uuid: "318814c64d7a4366a2f6a186a1298da9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39939 } }
I20260927 02:33:36.069875 1442 ts_tablet_manager.cc:1428] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 09da89815e9048cda1338c034c6cb579: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260927 02:33:36.158082 1288 tablet.cc:2369] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 09da89815e9048cda1338c034c6cb579: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:33:36.158550 1447 raft_consensus.cc:482] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 8e36a9146c5648709151cce5d91869c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:33:36.158730 1447 raft_consensus.cc:504] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 8e36a9146c5648709151cce5d91869c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e36a9146c5648709151cce5d91869c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36507 } } peers { permanent_uuid: "09da89815e9048cda1338c034c6cb579" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35647 } } peers { permanent_uuid: "318814c64d7a4366a2f6a186a1298da9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39939 } }
I20260927 02:33:36.159978 1447 leader_election.cc:288] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 8e36a9146c5648709151cce5d91869c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09da89815e9048cda1338c034c6cb579 (127.0.98.2:35647), 318814c64d7a4366a2f6a186a1298da9 (127.0.98.3:39939)
I20260927 02:33:36.164469 1449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:36.160216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35647 (local address 127.0.98.1:57161)
0927 02:33:36.160599 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:36.160612 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:36.160744 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:36.161094 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:36.161097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:36.161117 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:33:36.161263 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:36.161270 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:36.162128 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:36.162131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:36.163001 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:36.163010 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:36.164004 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:36.164008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:36.164105 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:36.164132 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 02:33:36.164218 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 02:33:36.164292 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":118,"threads_started":1}
I20260927 02:33:36.164978 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:36.160312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57161 (local address 127.0.98.2:35647)
0927 02:33:36.160526 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:36.160531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:36.160707 (+ 176us) server_negotiation.cc:407] Connection header received
0927 02:33:36.160907 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:36.160910 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:36.160993 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:36.161057 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:33:36.161442 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:36.162030 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:36.163119 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:36.163820 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:36.163933 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:36.163998 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:36.164393 (+ 395us) server_negotiation.cc:299] Negotiation successful
0927 02:33:36.164460 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260927 02:33:36.165055 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:36.160768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36077 (local address 127.0.98.3:39939)
0927 02:33:36.160893 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:36.160897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:36.161023 (+ 126us) server_negotiation.cc:407] Connection header received
0927 02:33:36.161104 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:36.161106 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:36.161149 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:36.161224 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:33:36.161606 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:36.162150 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:36.163948 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:36.164565 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:36.164661 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:36.164718 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:36.164863 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 02:33:36.164952 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 02:33:36.165061 1450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:36.160617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39939 (local address 127.0.98.1:36077)
0927 02:33:36.160893 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:36.160906 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:36.161027 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:36.161236 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:36.161238 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:36.161254 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:33:36.161490 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:36.161498 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:36.162253 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:36.162255 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:36.163784 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:36.163794 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:36.164665 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:36.164669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:36.164754 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:36.164773 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 02:33:36.164833 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 02:33:36.164895 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":95,"threads_started":1}
I20260927 02:33:36.165627 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d40e889b77e4a5ba5e4cddf3ae55b40" candidate_uuid: "8e36a9146c5648709151cce5d91869c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09da89815e9048cda1338c034c6cb579" is_pre_election: true
I20260927 02:33:36.165834 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d40e889b77e4a5ba5e4cddf3ae55b40" candidate_uuid: "8e36a9146c5648709151cce5d91869c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "318814c64d7a4366a2f6a186a1298da9" is_pre_election: true
I20260927 02:33:36.165951 1241 raft_consensus.cc:2454] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 09da89815e9048cda1338c034c6cb579 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e36a9146c5648709151cce5d91869c2 in term 0.
I20260927 02:33:36.166126 1375 raft_consensus.cc:2454] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 318814c64d7a4366a2f6a186a1298da9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e36a9146c5648709151cce5d91869c2 in term 0.
I20260927 02:33:36.166564 1042 leader_election.cc:302] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 8e36a9146c5648709151cce5d91869c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09da89815e9048cda1338c034c6cb579, 8e36a9146c5648709151cce5d91869c2; no voters:
I20260927 02:33:36.166939 1447 raft_consensus.cc:2781] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 8e36a9146c5648709151cce5d91869c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:33:36.167042 1447 raft_consensus.cc:482] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 8e36a9146c5648709151cce5d91869c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:33:36.167090 1447 raft_consensus.cc:3037] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 8e36a9146c5648709151cce5d91869c2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:33:36.178331 1447 raft_consensus.cc:504] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 8e36a9146c5648709151cce5d91869c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e36a9146c5648709151cce5d91869c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36507 } } peers { permanent_uuid: "09da89815e9048cda1338c034c6cb579" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35647 } } peers { permanent_uuid: "318814c64d7a4366a2f6a186a1298da9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39939 } }
I20260927 02:33:36.179199 1447 leader_election.cc:288] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 8e36a9146c5648709151cce5d91869c2 [CANDIDATE]: Term 1 election: Requested vote from peers 09da89815e9048cda1338c034c6cb579 (127.0.98.2:35647), 318814c64d7a4366a2f6a186a1298da9 (127.0.98.3:39939)
I20260927 02:33:36.179479 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d40e889b77e4a5ba5e4cddf3ae55b40" candidate_uuid: "8e36a9146c5648709151cce5d91869c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09da89815e9048cda1338c034c6cb579"
I20260927 02:33:36.179636 1241 raft_consensus.cc:3037] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 09da89815e9048cda1338c034c6cb579 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:33:36.180023 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d40e889b77e4a5ba5e4cddf3ae55b40" candidate_uuid: "8e36a9146c5648709151cce5d91869c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "318814c64d7a4366a2f6a186a1298da9"
I20260927 02:33:36.180156 1375 raft_consensus.cc:3037] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 318814c64d7a4366a2f6a186a1298da9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:33:36.181173 1241 raft_consensus.cc:2454] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 09da89815e9048cda1338c034c6cb579 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e36a9146c5648709151cce5d91869c2 in term 1.
I20260927 02:33:36.181468 1375 raft_consensus.cc:2454] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 318814c64d7a4366a2f6a186a1298da9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e36a9146c5648709151cce5d91869c2 in term 1.
I20260927 02:33:36.181610 1042 leader_election.cc:302] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 8e36a9146c5648709151cce5d91869c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09da89815e9048cda1338c034c6cb579, 8e36a9146c5648709151cce5d91869c2; no voters:
I20260927 02:33:36.181874 1447 raft_consensus.cc:2781] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 8e36a9146c5648709151cce5d91869c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:33:36.182096 1447 raft_consensus.cc:686] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 8e36a9146c5648709151cce5d91869c2 [term 1 LEADER]: Becoming Leader. State: Replica: 8e36a9146c5648709151cce5d91869c2, State: Running, Role: LEADER
I20260927 02:33:36.182672 1447 consensus_queue.cc:237] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 8e36a9146c5648709151cce5d91869c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e36a9146c5648709151cce5d91869c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36507 } } peers { permanent_uuid: "09da89815e9048cda1338c034c6cb579" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35647 } } peers { permanent_uuid: "318814c64d7a4366a2f6a186a1298da9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39939 } }
I20260927 02:33:36.186190 962 catalog_manager.cc:5495] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 8e36a9146c5648709151cce5d91869c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e36a9146c5648709151cce5d91869c2 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "8e36a9146c5648709151cce5d91869c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e36a9146c5648709151cce5d91869c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09da89815e9048cda1338c034c6cb579" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "318814c64d7a4366a2f6a186a1298da9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39939 } health_report { overall_health: UNKNOWN } } }
W20260927 02:33:36.205740 1154 tablet.cc:2369] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 8e36a9146c5648709151cce5d91869c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 02:33:36 dist-test-slave-grlk krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:33:36.236244 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:36.225733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57302 (local address 127.0.98.1:36507)
0927 02:33:36.225905 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:36.225910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:36.226023 (+ 113us) server_negotiation.cc:407] Connection header received
0927 02:33:36.226182 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:36.226186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:36.226241 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:36.226584 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:36.227278 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:36.228095 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:36.229163 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:36.229423 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:36.229522 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:36.232199 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:36.232217 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:36.232240 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:36.232349 (+ 109us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:36.234478 (+ 2129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:36.234972 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:36.234978 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:36.234979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:36.235080 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:36.235369 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:36.235372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:36.235373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:36.235641 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:36.235797 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:36.235992 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 02:33:36.236099 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260927 02:33:36.240119 1422 tablet.cc:2369] T 0d40e889b77e4a5ba5e4cddf3ae55b40 P 318814c64d7a4366a2f6a186a1298da9: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 02:33:36 dist-test-slave-grlk krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476416, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 02:33:36 dist-test-slave-grlk krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476416, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:33:36.291106 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:36.282552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57304 (local address 127.0.98.1:36507)
0927 02:33:36.282687 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:36.282691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:36.282779 (+ 88us) server_negotiation.cc:407] Connection header received
0927 02:33:36.282905 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:36.282908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:36.282951 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:36.283027 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:36.283819 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:36.284352 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:36.285461 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:36.285689 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:36.285767 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:36.288144 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:36.288148 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:36.288152 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:36.288180 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:36.289626 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:36.290011 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:36.290014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:36.290015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:36.290082 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:36.290275 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:36.290277 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:36.290278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:36.290424 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:36.290518 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:36.290820 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 02:33:36.290940 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Sep 27 02:33:36 dist-test-slave-grlk krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 02:33:36 dist-test-slave-grlk krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:33:36.394768 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:36.386657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57306 (local address 127.0.98.1:36507)
0927 02:33:36.386796 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:36.386800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:36.386925 (+ 125us) server_negotiation.cc:407] Connection header received
0927 02:33:36.387071 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:36.387074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:36.387117 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:36.387185 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:36.388103 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:36.388623 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:36.389520 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:36.389727 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:36.389800 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:36.392113 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:36.392116 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:36.392118 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:36.392140 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:36.393484 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:36.393862 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:36.393865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:36.393866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:36.393929 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:36.394116 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:36.394118 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:36.394119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:36.394254 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:36.394338 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:36.394515 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 02:33:36.394603 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260927 02:33:36.396049 1067 tablet_service.cc:2908] Rejecting scan request for tablet 0d40e889b77e4a5ba5e4cddf3ae55b40: Uninitialized: clean time has not yet been initialized
I20260927 02:33:36.397841 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 1024
I20260927 02:33:36.403019 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 1158
I20260927 02:33:36.407675 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 1292
I20260927 02:33:36.412174 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 930
2026-09-27T02:33:36Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3470 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:33:37 dist-test-slave-grlk krb5kdc[1465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:33:37 dist-test-slave-grlk krb5kdc[1465](info): set up 2 sockets
Sep 27 02:33:37 dist-test-slave-grlk krb5kdc[1465](info): commencing operation
krb5kdc: starting...
W20260927 02:33:38.728825 392 mini_kdc.cc:121] Time spent starting KDC: real 2.295s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260927 02:33:38.859397 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.130s user 0.001s sys 0.000s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260927 02:33:39.000739 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.141s user 0.001s sys 0.000s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260927 02:33:39.202046 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.201s user 0.001s sys 0.000s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/krb5kdc/test-user.keytab.
W20260927 02:33:39.856956 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.655s user 0.002s sys 0.000s
Sep 27 02:33:39 dist-test-slave-grlk krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:33:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:33:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
W20260927 02:33:40.347225 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.62: real 0.448s user 0.000s sys 0.003s
I20260927 02:33:40.347713 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43451
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:33:40.453505 1478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:33:40.453758 1478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:33:40.453796 1478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:33:40.457717 1478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:33:40.457764 1478 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 02:33:40.457780 1478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:33:40.457798 1478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:33:40.457811 1478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:33:40.457834 1478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:33:40.462934 1478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43451
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:33:40.463812 1478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:33:40.464702 1478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:33:40.472566 1486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:40.472613 1483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:40.472697 1484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:40.472671 1478 server_base.cc:1034] running on GCE node
I20260927 02:33:40.473366 1478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:33:40.474437 1478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:33:40.475617 1478 hybrid_clock.cc:648] HybridClock initialized: now 1790476420475604 us; error 25 us; skew 500 ppm
Sep 27 02:33:40 dist-test-slave-grlk krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:33:40.478236 1478 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:33:40.479063 1478 webserver.cc:460] Webserver started at http://127.0.98.62:35865/ using document root <none> and password file <none>
I20260927 02:33:40.479584 1478 fs_manager.cc:359] Metadata directory not provided
I20260927 02:33:40.479648 1478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:33:40.479838 1478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:33:40.482059 1478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "0c09af2b5fc64aa3bf8204902b896b0a"
format_stamp: "Formatted at 2026-09-27 02:33:40 on dist-test-slave-grlk"
server_key: "ea07cbb30e2ecc88f994e65468aaf54f"
server_key_iv: "409626d068732ad9d1cc5f952a8f3ed1"
server_key_version: "encryptionkey@0"
I20260927 02:33:40.482676 1478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "0c09af2b5fc64aa3bf8204902b896b0a"
format_stamp: "Formatted at 2026-09-27 02:33:40 on dist-test-slave-grlk"
server_key: "ea07cbb30e2ecc88f994e65468aaf54f"
server_key_iv: "409626d068732ad9d1cc5f952a8f3ed1"
server_key_version: "encryptionkey@0"
I20260927 02:33:40.486315 1478 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:33:40.488677 1493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:40.489869 1478 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:33:40.489977 1478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "0c09af2b5fc64aa3bf8204902b896b0a"
format_stamp: "Formatted at 2026-09-27 02:33:40 on dist-test-slave-grlk"
server_key: "ea07cbb30e2ecc88f994e65468aaf54f"
server_key_iv: "409626d068732ad9d1cc5f952a8f3ed1"
server_key_version: "encryptionkey@0"
I20260927 02:33:40.490068 1478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:33:40.502477 1478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:33:40.505388 1478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:33:40.505558 1478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:33:40.512508 1478 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43451
I20260927 02:33:40.512750 1555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43451 every 8 connection(s)
I20260927 02:33:40.513592 1478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:33:40.516289 1556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:33:40.521747 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c09af2b5fc64aa3bf8204902b896b0a: Bootstrap starting.
I20260927 02:33:40.523149 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 1478
I20260927 02:33:40.523254 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:33:40.523499 392 external_mini_cluster.cc:1428] Setting key c02de1992404e6a2d3becc7e4280df65
I20260927 02:33:40.524343 1556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c09af2b5fc64aa3bf8204902b896b0a: Neither blocks nor log segments found. Creating new log.
I20260927 02:33:40.525249 1556 log.cc:824] T 00000000000000000000000000000000 P 0c09af2b5fc64aa3bf8204902b896b0a: Log is configured to *not* fsync() on all Append() calls
I20260927 02:33:40.526981 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c09af2b5fc64aa3bf8204902b896b0a: No bootstrap required, opened a new log
I20260927 02:33:40.530238 1556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c09af2b5fc64aa3bf8204902b896b0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c09af2b5fc64aa3bf8204902b896b0a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43451 } }
I20260927 02:33:40.530457 1556 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c09af2b5fc64aa3bf8204902b896b0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:33:40.530511 1556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c09af2b5fc64aa3bf8204902b896b0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c09af2b5fc64aa3bf8204902b896b0a, State: Initialized, Role: FOLLOWER
I20260927 02:33:40.530932 1556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c09af2b5fc64aa3bf8204902b896b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c09af2b5fc64aa3bf8204902b896b0a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43451 } }
I20260927 02:33:40.531062 1556 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c09af2b5fc64aa3bf8204902b896b0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:33:40.531126 1556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c09af2b5fc64aa3bf8204902b896b0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
Sep 27 02:33:40 dist-test-slave-grlk krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:33:40.531221 1556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c09af2b5fc64aa3bf8204902b896b0a [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:33:40.532178 1556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c09af2b5fc64aa3bf8204902b896b0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c09af2b5fc64aa3bf8204902b896b0a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43451 } }
I20260927 02:33:40.532485 1556 leader_election.cc:302] T 00000000000000000000000000000000 P 0c09af2b5fc64aa3bf8204902b896b0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c09af2b5fc64aa3bf8204902b896b0a; no voters:
I20260927 02:33:40.532752 1556 leader_election.cc:288] T 00000000000000000000000000000000 P 0c09af2b5fc64aa3bf8204902b896b0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:33:40.532871 1561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c09af2b5fc64aa3bf8204902b896b0a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:33:40.533139 1561 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c09af2b5fc64aa3bf8204902b896b0a [term 1 LEADER]: Becoming Leader. State: Replica: 0c09af2b5fc64aa3bf8204902b896b0a, State: Running, Role: LEADER
I20260927 02:33:40.533489 1561 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c09af2b5fc64aa3bf8204902b896b0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c09af2b5fc64aa3bf8204902b896b0a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43451 } }
I20260927 02:33:40.533984 1556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c09af2b5fc64aa3bf8204902b896b0a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:33:40.534566 1561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c09af2b5fc64aa3bf8204902b896b0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c09af2b5fc64aa3bf8204902b896b0a. Latest consensus state: current_term: 1 leader_uuid: "0c09af2b5fc64aa3bf8204902b896b0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c09af2b5fc64aa3bf8204902b896b0a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43451 } } }
I20260927 02:33:40.535264 1561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c09af2b5fc64aa3bf8204902b896b0a [sys.catalog]: This master's current role is: LEADER
I20260927 02:33:40.535163 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c09af2b5fc64aa3bf8204902b896b0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c09af2b5fc64aa3bf8204902b896b0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c09af2b5fc64aa3bf8204902b896b0a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43451 } } }
I20260927 02:33:40.535535 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c09af2b5fc64aa3bf8204902b896b0a [sys.catalog]: This master's current role is: LEADER
I20260927 02:33:40.535815 1570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:33:40.537396 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:40.524566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56892 (local address 127.0.98.62:43451)
0927 02:33:40.524995 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:40.525010 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:40.525043 (+ 33us) server_negotiation.cc:407] Connection header received
0927 02:33:40.525685 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:40.525702 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:40.526007 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:40.526305 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:40.527037 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:40.527988 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:40.528799 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:40.529306 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:40.529444 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:40.531767 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:40.531772 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:40.531783 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:40.531807 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:40.533638 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:40.534091 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:40.534096 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:40.534102 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:40.534232 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:40.534566 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:40.534569 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:40.534571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:40.535097 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:40.535296 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:40.536150 (+ 854us) server_negotiation.cc:299] Negotiation successful
0927 02:33:40.536462 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":125,"threads_started":1}
I20260927 02:33:40.538699 1570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:33:40.543726 1570 catalog_manager.cc:1347] Generated new cluster ID: 082a0047912244dc93d2b7415369ab3d
I20260927 02:33:40.543787 1570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:33:40.557717 1570 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:33:40.559000 1570 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:33:40.570139 1570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c09af2b5fc64aa3bf8204902b896b0a: Generated new TSK 0
I20260927 02:33:40.570895 1570 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:33:40.603673 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43451
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:33:40.715201 1582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:33:40.715435 1582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:33:40.715492 1582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:33:40.719434 1582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:33:40.719480 1582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:33:40.719532 1582 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:33:40.719584 1582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:33:40.724438 1582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:33:40.725358 1582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:33:40.726245 1582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:33:40.734805 1590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:40.734789 1588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:40.734869 1582 server_base.cc:1034] running on GCE node
W20260927 02:33:40.734774 1587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:40.735474 1582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:33:40.735991 1582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:33:40.737133 1582 hybrid_clock.cc:648] HybridClock initialized: now 1790476420737116 us; error 31 us; skew 500 ppm
Sep 27 02:33:40 dist-test-slave-grlk krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:33:40.739630 1582 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:33:40.740408 1582 webserver.cc:460] Webserver started at http://127.0.98.1:45491/ using document root <none> and password file <none>
I20260927 02:33:40.740887 1582 fs_manager.cc:359] Metadata directory not provided
I20260927 02:33:40.740972 1582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:33:40.741166 1582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:33:40.743016 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-0/data/instance:
uuid: "3e020b8f631243e2a93860e0fb7bdff7"
format_stamp: "Formatted at 2026-09-27 02:33:40 on dist-test-slave-grlk"
server_key: "aa13111fe126ac5e26b7ed34dae2b7db"
server_key_iv: "e4fea5df586e66bf99128b417ce36bdf"
server_key_version: "encryptionkey@0"
I20260927 02:33:40.743511 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3e020b8f631243e2a93860e0fb7bdff7"
format_stamp: "Formatted at 2026-09-27 02:33:40 on dist-test-slave-grlk"
server_key: "aa13111fe126ac5e26b7ed34dae2b7db"
server_key_iv: "e4fea5df586e66bf99128b417ce36bdf"
server_key_version: "encryptionkey@0"
I20260927 02:33:40.746840 1582 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:33:40.748955 1597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:40.750005 1582 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:33:40.750113 1582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-0/wal
uuid: "3e020b8f631243e2a93860e0fb7bdff7"
format_stamp: "Formatted at 2026-09-27 02:33:40 on dist-test-slave-grlk"
server_key: "aa13111fe126ac5e26b7ed34dae2b7db"
server_key_iv: "e4fea5df586e66bf99128b417ce36bdf"
server_key_version: "encryptionkey@0"
I20260927 02:33:40.750195 1582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:33:40.769348 1582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:33:40.772127 1582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:33:40.772282 1582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:33:40.773628 1582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:33:40.773679 1582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:40.773729 1582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:33:40.773754 1582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 02:33:40 dist-test-slave-grlk krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:33:40.783301 1582 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35549
I20260927 02:33:40.783361 1712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35549 every 8 connection(s)
I20260927 02:33:40.784087 1582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:33:40.786892 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:40.774284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59997 (local address 127.0.98.62:43451)
0927 02:33:40.774383 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:40.774387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:40.775361 (+ 974us) server_negotiation.cc:407] Connection header received
0927 02:33:40.776253 (+ 892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:40.776257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:40.776315 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:40.776403 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:40.777637 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:40.778306 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:40.779964 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:40.780250 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:40.780338 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:40.783540 (+ 3202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:40.783544 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:40.783546 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:40.783569 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:40.785273 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:40.785689 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:40.785692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:40.785694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:40.785778 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:40.786021 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:40.786024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:40.786026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:40.786230 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:40.786311 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:40.786683 (+ 372us) server_negotiation.cc:299] Negotiation successful
0927 02:33:40.786775 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 02:33:40.787691 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:40.774813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43451 (local address 127.0.98.1:59997)
0927 02:33:40.775230 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:40.775267 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:40.776039 (+ 772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:40.776648 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:40.776657 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:40.777056 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:33:40.777479 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:40.777492 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:40.778406 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:40.778409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:40.779815 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:40.779826 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:40.780428 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:40.780433 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:40.780575 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:40.781685 (+ 1110us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:33:40.781706 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:33:40.783358 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:33:40.785373 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:40.785377 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:40.785386 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:40.785606 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:40.785852 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:40.785854 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:40.785855 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:40.785957 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:40.786314 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:33:40.786318 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:33:40.786509 (+ 191us) client_negotiation.cc:769] Sending connection context
0927 02:33:40.786647 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 02:33:40.786859 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":120,"threads_started":1}
I20260927 02:33:40.788721 1713 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43451
I20260927 02:33:40.788849 1713 heartbeater.cc:464] Registering TS with master...
I20260927 02:33:40.789418 1713 heartbeater.cc:511] Master 127.0.98.62:43451 requested a full tablet report, sending...
I20260927 02:33:40.789456 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 1582
I20260927 02:33:40.789564 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:33:40.789814 392 external_mini_cluster.cc:1428] Setting key 80393b35cb0c86740c9dc71ef0c89df1
I20260927 02:33:40.790969 1509 ts_manager.cc:194] Registered new tserver with Master: 3e020b8f631243e2a93860e0fb7bdff7 (127.0.98.1:35549)
I20260927 02:33:40.792217 1509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59997
I20260927 02:33:40.797741 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:40.790961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38447 (local address 127.0.98.62:43451)
0927 02:33:40.791060 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:40.791064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:40.791288 (+ 224us) server_negotiation.cc:407] Connection header received
0927 02:33:40.791383 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:40.791386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:40.791424 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:40.791496 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:40.791924 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:40.792605 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:40.793783 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:40.794024 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:40.794111 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:40.795116 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:40.795118 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:40.795120 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:40.795138 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:40.796342 (+ 1204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:40.796762 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:40.796764 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:40.796765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:40.796824 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:40.797092 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:40.797095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:40.797096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:40.797235 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:40.797302 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:40.797555 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 02:33:40.797639 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 02:33:40.797838 1719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:40.790830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43451 (local address 127.0.98.1:38447)
0927 02:33:40.791202 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:40.791215 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:40.791306 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:40.791529 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:40.791531 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:40.791710 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:33:40.791817 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:40.791824 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:40.792744 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:40.792747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:40.793685 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:40.793695 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:40.794125 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:40.794129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:40.794218 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:40.794595 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:33:40.794618 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:33:40.794996 (+ 378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:33:40.796439 (+ 1443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:40.796443 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:40.796445 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:40.796680 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:40.796896 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:40.796900 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:40.796901 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:40.797016 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:40.797316 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:33:40.797319 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:33:40.797411 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 02:33:40.797513 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 02:33:40.797651 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":25,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:33:40.814898 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43451
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:33:40.918452 1722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:33:40.918701 1722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:33:40.918743 1722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:33:40.922748 1722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:33:40.922809 1722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:33:40.922852 1722 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:33:40.922902 1722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:33:40.927983 1722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:33:40.929005 1722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:33:40.929982 1722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:33:40.939024 1728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:40.939127 1730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:40.939139 1727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:40.939327 1722 server_base.cc:1034] running on GCE node
I20260927 02:33:40.939677 1722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:33:40.940266 1722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:33:40.941430 1722 hybrid_clock.cc:648] HybridClock initialized: now 1790476420941397 us; error 37 us; skew 500 ppm
Sep 27 02:33:40 dist-test-slave-grlk krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:33:40.943886 1722 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:33:40.944811 1722 webserver.cc:460] Webserver started at http://127.0.98.2:46425/ using document root <none> and password file <none>
I20260927 02:33:40.945398 1722 fs_manager.cc:359] Metadata directory not provided
I20260927 02:33:40.945459 1722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:33:40.945626 1722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:33:40.947613 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-1/data/instance:
uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0"
format_stamp: "Formatted at 2026-09-27 02:33:40 on dist-test-slave-grlk"
server_key: "a03f3489546653c9ac7daacb0ead8123"
server_key_iv: "b267e305762a80d73f8f5d4fea01f61c"
server_key_version: "encryptionkey@0"
I20260927 02:33:40.948093 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0"
format_stamp: "Formatted at 2026-09-27 02:33:40 on dist-test-slave-grlk"
server_key: "a03f3489546653c9ac7daacb0ead8123"
server_key_iv: "b267e305762a80d73f8f5d4fea01f61c"
server_key_version: "encryptionkey@0"
I20260927 02:33:40.951735 1722 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 02:33:40.954161 1737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:40.955379 1722 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:33:40.955470 1722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-1/wal
uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0"
format_stamp: "Formatted at 2026-09-27 02:33:40 on dist-test-slave-grlk"
server_key: "a03f3489546653c9ac7daacb0ead8123"
server_key_iv: "b267e305762a80d73f8f5d4fea01f61c"
server_key_version: "encryptionkey@0"
I20260927 02:33:40.955551 1722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:33:40.974112 1722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:33:40.976643 1722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:33:40.976781 1722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:33:40.978111 1722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:33:40.978170 1722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:40.978223 1722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:33:40.978250 1722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 02:33:40 dist-test-slave-grlk krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:33:40.987617 1722 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33553
I20260927 02:33:40.987665 1852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33553 every 8 connection(s)
I20260927 02:33:40.988507 1722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:33:40.989771 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 1722
I20260927 02:33:40.989902 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:33:40.990190 392 external_mini_cluster.cc:1428] Setting key 8a151ea37e4c79e3865780e12487ab09
I20260927 02:33:40.990643 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:40.978848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55657 (local address 127.0.98.62:43451)
0927 02:33:40.978983 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:40.978988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:40.979929 (+ 941us) server_negotiation.cc:407] Connection header received
0927 02:33:40.981026 (+ 1097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:40.981030 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:40.981089 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:40.981192 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:40.982389 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:40.983180 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:40.984368 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:40.984594 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:40.984680 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:40.986976 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:40.986980 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:40.986983 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:40.987010 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:40.988457 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:40.989027 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:40.989031 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:40.989032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:40.989125 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:40.989494 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:40.989497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:40.989498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:40.989731 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:40.989810 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:40.990407 (+ 597us) server_negotiation.cc:299] Negotiation successful
0927 02:33:40.990498 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 02:33:40.991496 1746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:40.979236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43451 (local address 127.0.98.2:55657)
0927 02:33:40.979764 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:40.979814 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:40.980777 (+ 963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:40.981366 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:40.981375 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:40.981742 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:33:40.982262 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:40.982276 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:40.983304 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:40.983308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:40.984260 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:40.984269 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:40.984732 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:40.984736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:40.984883 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:40.985416 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:33:40.985431 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:33:40.986827 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:33:40.988556 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:40.988561 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:40.988574 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:40.988861 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:40.989235 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:40.989238 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:40.989240 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:40.989397 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:40.989845 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:33:40.989851 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:33:40.990165 (+ 314us) client_negotiation.cc:769] Sending connection context
0927 02:33:40.990360 (+ 195us) client_negotiation.cc:240] Negotiation successful
0927 02:33:40.990667 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":147,"threads_started":1}
I20260927 02:33:40.992762 1853 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43451
I20260927 02:33:40.992893 1853 heartbeater.cc:464] Registering TS with master...
I20260927 02:33:40.993491 1853 heartbeater.cc:511] Master 127.0.98.62:43451 requested a full tablet report, sending...
I20260927 02:33:40.994850 1509 ts_manager.cc:194] Registered new tserver with Master: e0e08aa3ca404cea99e4b5da45d4a9b0 (127.0.98.2:33553)
I20260927 02:33:40.995514 1509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55657
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 02:33:41.001808 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:40.994683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55321 (local address 127.0.98.62:43451)
0927 02:33:40.994773 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:40.994776 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:40.995064 (+ 288us) server_negotiation.cc:407] Connection header received
0927 02:33:40.995180 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:40.995183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:40.995224 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:40.995294 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:40.995801 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:40.996568 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:40.997567 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:40.997890 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:40.997999 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:40.998891 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:40.998895 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:40.998898 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:40.998920 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:41.000217 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:41.000612 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:41.000614 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:41.000615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:41.000678 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:41.001047 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:41.001049 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:41.001051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:41.001242 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:41.001317 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:41.001589 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 02:33:41.001695 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 02:33:41.001818 1860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:40.994660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43451 (local address 127.0.98.2:55321)
0927 02:33:40.994983 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:40.994998 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:40.995091 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:40.995385 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:40.995388 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:40.995581 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:33:40.995686 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:40.995693 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:40.996699 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:40.996702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:40.997458 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:40.997466 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:40.997989 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:40.997991 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:40.998052 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:40.998462 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:33:40.998472 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:33:40.998766 (+ 294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:33:41.000313 (+ 1547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:41.000316 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:41.000318 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:41.000532 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:41.000773 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:41.000775 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:41.000777 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:41.000915 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:41.001337 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:33:41.001339 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:33:41.001431 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 02:33:41.001527 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 02:33:41.001644 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":90,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:33:41.015383 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43451
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:33:41.122291 1862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:33:41.122514 1862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:33:41.122570 1862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:33:41.126509 1862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:33:41.126569 1862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:33:41.126621 1862 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:33:41.126672 1862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:33:41.131785 1862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:33:41.132730 1862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:33:41.133718 1862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:33:41.142436 1868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:41.142480 1867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:41.142549 1862 server_base.cc:1034] running on GCE node
W20260927 02:33:41.142457 1870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:41.143052 1862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:33:41.143635 1862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:33:41.144793 1862 hybrid_clock.cc:648] HybridClock initialized: now 1790476421144777 us; error 29 us; skew 500 ppm
Sep 27 02:33:41 dist-test-slave-grlk krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476421, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:33:41.147192 1862 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:33:41.147989 1862 webserver.cc:460] Webserver started at http://127.0.98.3:37377/ using document root <none> and password file <none>
I20260927 02:33:41.148502 1862 fs_manager.cc:359] Metadata directory not provided
I20260927 02:33:41.148563 1862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:33:41.148737 1862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:33:41.150767 1862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-2/data/instance:
uuid: "e62fb61a7e20421caccba2f459dd16fb"
format_stamp: "Formatted at 2026-09-27 02:33:41 on dist-test-slave-grlk"
server_key: "4e8414b9c3fec6013aac4c7a0249ef5d"
server_key_iv: "bc74162e34704b847e5a19f1011de3f8"
server_key_version: "encryptionkey@0"
I20260927 02:33:41.151284 1862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e62fb61a7e20421caccba2f459dd16fb"
format_stamp: "Formatted at 2026-09-27 02:33:41 on dist-test-slave-grlk"
server_key: "4e8414b9c3fec6013aac4c7a0249ef5d"
server_key_iv: "bc74162e34704b847e5a19f1011de3f8"
server_key_version: "encryptionkey@0"
I20260927 02:33:41.154980 1862 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:33:41.157222 1877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:41.158361 1862 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 02:33:41.158465 1862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-2/wal
uuid: "e62fb61a7e20421caccba2f459dd16fb"
format_stamp: "Formatted at 2026-09-27 02:33:41 on dist-test-slave-grlk"
server_key: "4e8414b9c3fec6013aac4c7a0249ef5d"
server_key_iv: "bc74162e34704b847e5a19f1011de3f8"
server_key_version: "encryptionkey@0"
I20260927 02:33:41.158553 1862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:33:41.171295 1862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:33:41.174307 1862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:33:41.174499 1862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:33:41.175849 1862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:33:41.175897 1862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:41.175948 1862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:33:41.175976 1862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 02:33:41 dist-test-slave-grlk krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476421, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:33:41.185679 1862 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44251
I20260927 02:33:41.185722 1992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44251 every 8 connection(s)
I20260927 02:33:41.186473 1862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:33:41.188122 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:41.176494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36611 (local address 127.0.98.62:43451)
0927 02:33:41.176628 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:41.176633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:41.177554 (+ 921us) server_negotiation.cc:407] Connection header received
0927 02:33:41.178709 (+ 1155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:41.178713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:41.178795 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:41.178882 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:41.180149 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:41.180961 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:41.182104 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:41.182437 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:41.182544 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:41.184792 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:41.184795 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:41.184798 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:41.184823 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:41.186284 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:41.186766 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:41.186770 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:41.186772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:41.186853 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:41.187193 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:41.187197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:41.187199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:41.187368 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:41.187451 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:41.187899 (+ 448us) server_negotiation.cc:299] Negotiation successful
0927 02:33:41.188003 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 02:33:41.188874 1886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:41.176883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43451 (local address 127.0.98.3:36611)
0927 02:33:41.177378 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:41.177434 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:41.178479 (+ 1045us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:41.179059 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:41.179066 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:41.179477 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:33:41.179975 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:41.179993 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:41.181081 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:41.181084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:41.181978 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:41.181986 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:41.182590 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:41.182595 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:41.182772 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:41.183302 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:33:41.183317 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:33:41.184655 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:33:41.186383 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:41.186388 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:41.186401 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:41.186683 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:41.186949 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:41.186952 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:41.186954 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:41.187111 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:41.187461 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:33:41.187465 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:33:41.187667 (+ 202us) client_negotiation.cc:769] Sending connection context
0927 02:33:41.187809 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 02:33:41.188028 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":104,"threads_started":1}
I20260927 02:33:41.189946 1993 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43451
I20260927 02:33:41.190069 1993 heartbeater.cc:464] Registering TS with master...
I20260927 02:33:41.190460 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 1862
I20260927 02:33:41.190500 1993 heartbeater.cc:511] Master 127.0.98.62:43451 requested a full tablet report, sending...
I20260927 02:33:41.190565 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:33:41.190858 392 external_mini_cluster.cc:1428] Setting key 64ae3e93e9d4ec2b108666502863c577
I20260927 02:33:41.191484 1510 ts_manager.cc:194] Registered new tserver with Master: e62fb61a7e20421caccba2f459dd16fb (127.0.98.3:44251)
I20260927 02:33:41.192185 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36611
I20260927 02:33:41.193487 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:33:41.198632 1999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:41.192095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43451 (local address 127.0.98.3:54409)
0927 02:33:41.192371 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:41.192384 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:41.192467 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:41.192672 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:41.192674 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:41.192851 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:33:41.192988 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:41.192996 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:41.193963 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:41.193965 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:41.194561 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:41.194569 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:41.195046 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:41.195049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:41.195139 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:41.195556 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:33:41.195569 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:33:41.195931 (+ 362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:33:41.197384 (+ 1453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:41.197387 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:41.197389 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:41.197612 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:41.197842 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:41.197844 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:41.197847 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:41.197927 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:41.198224 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:33:41.198227 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:33:41.198314 (+ 87us) client_negotiation.cc:769] Sending connection context
0927 02:33:41.198409 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 02:33:41.198501 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":87,"threads_started":1}
I20260927 02:33:41.198730 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:41.192175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54409 (local address 127.0.98.62:43451)
0927 02:33:41.192276 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:41.192280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:41.192437 (+ 157us) server_negotiation.cc:407] Connection header received
0927 02:33:41.192549 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:41.192551 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:41.192597 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:41.192658 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:41.193113 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:41.193837 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:41.194657 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:41.194947 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:41.195048 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:41.196057 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:41.196060 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:41.196061 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:41.196080 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:41.197287 (+ 1207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:41.197696 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:41.197699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:41.197700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:41.197767 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:41.198005 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:41.198007 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:41.198008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:41.198147 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:41.198213 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:41.198475 (+ 262us) server_negotiation.cc:299] Negotiation successful
0927 02:33:41.198575 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
Sep 27 02:33:41 dist-test-slave-grlk krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476421, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 02:33:41 dist-test-slave-grlk krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476421, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:33:41.216508 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:41.209833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45264 (local address 127.0.98.62:43451)
0927 02:33:41.209941 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:41.209945 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:41.209955 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:33:41.209990 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:41.209992 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:41.210026 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:41.210089 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:41.210677 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:41.211153 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:41.211911 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:41.212112 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:41.212178 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:41.214254 (+ 2076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:41.214257 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:41.214259 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:41.214280 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:41.215429 (+ 1149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:41.215723 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:41.215725 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:41.215726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:41.215784 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:41.215944 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:41.215946 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:41.215947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:41.216083 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:41.216170 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:41.216323 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 02:33:41.216398 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 02:33:41.218914 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45264:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:33:41.221916 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:33:41.234687 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:41.230055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45054 (local address 127.0.98.3:44251)
0927 02:33:41.230438 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:41.230448 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:41.230468 (+ 20us) server_negotiation.cc:407] Connection header received
0927 02:33:41.230536 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:41.230539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:41.230676 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:41.230794 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:33:41.230996 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:41.231711 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:41.233222 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:41.234044 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:41.234149 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:41.234311 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:41.234444 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 02:33:41.234534 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":102,"threads_started":1}
I20260927 02:33:41.234678 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:41.229169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44251 (local address 127.0.0.1:45054)
0927 02:33:41.230356 (+ 1187us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:41.230370 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:41.230449 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:41.230785 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:41.230788 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:41.230804 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:33:41.230885 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:41.230891 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:41.231809 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:41.231821 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:41.233113 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:41.233123 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:41.234162 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:41.234165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:41.234272 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:41.234310 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 02:33:41.234416 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 02:33:41.234526 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":68,"threads_started":1}
I20260927 02:33:41.235358 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:41.229226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37272 (local address 127.0.98.2:33553)
0927 02:33:41.229523 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:41.229533 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:41.229965 (+ 432us) server_negotiation.cc:407] Connection header received
0927 02:33:41.230047 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:41.230052 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:41.230271 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:41.230432 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:33:41.230841 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:41.231991 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:41.233393 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:41.234158 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:41.234242 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:41.234364 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:41.235130 (+ 766us) server_negotiation.cc:299] Negotiation successful
0927 02:33:41.235227 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":106,"threads_started":1}
I20260927 02:33:41.235378 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:41.229702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35549 (local address 127.0.0.1:48260)
0927 02:33:41.230383 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:41.230395 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:41.230470 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:41.230938 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:41.230941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:41.230955 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:33:41.231019 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:41.231024 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:41.232235 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:41.232238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:41.234001 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:41.234012 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:41.235043 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:41.235047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:41.235134 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:41.235155 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 02:33:41.235216 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 02:33:41.235271 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"mutex_wait_us":2,"thread_start_us":75,"threads_started":1}
I20260927 02:33:41.235538 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:41.229887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48260 (local address 127.0.98.1:35549)
0927 02:33:41.230253 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:41.230263 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:41.230556 (+ 293us) server_negotiation.cc:407] Connection header received
0927 02:33:41.230642 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:41.230647 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:41.230830 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:41.230964 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:33:41.231101 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:41.232142 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:41.234135 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:41.234932 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:41.235031 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:41.235285 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:41.235331 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 02:33:41.235411 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":94,"threads_started":1}
I20260927 02:33:41.235724 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:41.229064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33553 (local address 127.0.0.1:37272)
0927 02:33:41.229661 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:41.229707 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:41.229831 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:41.230587 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:41.230592 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:41.230612 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:33:41.230720 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:41.230725 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:41.232104 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:41.232107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:41.233267 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:41.233275 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:41.234422 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:41.234425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:41.234523 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:41.234544 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 02:33:41.234611 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 02:33:41.234677 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"mutex_wait_us":41,"thread_start_us":49,"threads_started":1}
I20260927 02:33:41.237752 1927 tablet_service.cc:1467] Processing CreateTablet for tablet e8fe49295d774ba49452444a2fd07e2a (DEFAULT_TABLE table=test-table [id=df0ff8ccf7814024828d495b3d478c49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:33:41.237752 1647 tablet_service.cc:1467] Processing CreateTablet for tablet e8fe49295d774ba49452444a2fd07e2a (DEFAULT_TABLE table=test-table [id=df0ff8ccf7814024828d495b3d478c49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:33:41.238013 1787 tablet_service.cc:1467] Processing CreateTablet for tablet e8fe49295d774ba49452444a2fd07e2a (DEFAULT_TABLE table=test-table [id=df0ff8ccf7814024828d495b3d478c49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:33:41.238780 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8fe49295d774ba49452444a2fd07e2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:33:41.238780 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8fe49295d774ba49452444a2fd07e2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:33:41.238802 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8fe49295d774ba49452444a2fd07e2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:33:41.245028 2016 tablet_bootstrap.cc:492] T e8fe49295d774ba49452444a2fd07e2a P e0e08aa3ca404cea99e4b5da45d4a9b0: Bootstrap starting.
I20260927 02:33:41.246419 2018 tablet_bootstrap.cc:492] T e8fe49295d774ba49452444a2fd07e2a P 3e020b8f631243e2a93860e0fb7bdff7: Bootstrap starting.
I20260927 02:33:41.246419 2017 tablet_bootstrap.cc:492] T e8fe49295d774ba49452444a2fd07e2a P e62fb61a7e20421caccba2f459dd16fb: Bootstrap starting.
I20260927 02:33:41.247076 2016 tablet_bootstrap.cc:654] T e8fe49295d774ba49452444a2fd07e2a P e0e08aa3ca404cea99e4b5da45d4a9b0: Neither blocks nor log segments found. Creating new log.
I20260927 02:33:41.247766 2016 log.cc:824] T e8fe49295d774ba49452444a2fd07e2a P e0e08aa3ca404cea99e4b5da45d4a9b0: Log is configured to *not* fsync() on all Append() calls
I20260927 02:33:41.248377 2017 tablet_bootstrap.cc:654] T e8fe49295d774ba49452444a2fd07e2a P e62fb61a7e20421caccba2f459dd16fb: Neither blocks nor log segments found. Creating new log.
I20260927 02:33:41.248377 2018 tablet_bootstrap.cc:654] T e8fe49295d774ba49452444a2fd07e2a P 3e020b8f631243e2a93860e0fb7bdff7: Neither blocks nor log segments found. Creating new log.
I20260927 02:33:41.249184 2017 log.cc:824] T e8fe49295d774ba49452444a2fd07e2a P e62fb61a7e20421caccba2f459dd16fb: Log is configured to *not* fsync() on all Append() calls
I20260927 02:33:41.249179 2018 log.cc:824] T e8fe49295d774ba49452444a2fd07e2a P 3e020b8f631243e2a93860e0fb7bdff7: Log is configured to *not* fsync() on all Append() calls
I20260927 02:33:41.249621 2016 tablet_bootstrap.cc:492] T e8fe49295d774ba49452444a2fd07e2a P e0e08aa3ca404cea99e4b5da45d4a9b0: No bootstrap required, opened a new log
I20260927 02:33:41.249807 2016 ts_tablet_manager.cc:1397] T e8fe49295d774ba49452444a2fd07e2a P e0e08aa3ca404cea99e4b5da45d4a9b0: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260927 02:33:41.250751 2017 tablet_bootstrap.cc:492] T e8fe49295d774ba49452444a2fd07e2a P e62fb61a7e20421caccba2f459dd16fb: No bootstrap required, opened a new log
I20260927 02:33:41.250911 2017 ts_tablet_manager.cc:1397] T e8fe49295d774ba49452444a2fd07e2a P e62fb61a7e20421caccba2f459dd16fb: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 02:33:41.251012 2018 tablet_bootstrap.cc:492] T e8fe49295d774ba49452444a2fd07e2a P 3e020b8f631243e2a93860e0fb7bdff7: No bootstrap required, opened a new log
I20260927 02:33:41.251188 2018 ts_tablet_manager.cc:1397] T e8fe49295d774ba49452444a2fd07e2a P 3e020b8f631243e2a93860e0fb7bdff7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 02:33:41.253590 2017 raft_consensus.cc:348] T e8fe49295d774ba49452444a2fd07e2a P e62fb61a7e20421caccba2f459dd16fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33553 } } peers { permanent_uuid: "e62fb61a7e20421caccba2f459dd16fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 } } peers { permanent_uuid: "3e020b8f631243e2a93860e0fb7bdff7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35549 } }
I20260927 02:33:41.253582 2016 raft_consensus.cc:348] T e8fe49295d774ba49452444a2fd07e2a P e0e08aa3ca404cea99e4b5da45d4a9b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33553 } } peers { permanent_uuid: "e62fb61a7e20421caccba2f459dd16fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 } } peers { permanent_uuid: "3e020b8f631243e2a93860e0fb7bdff7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35549 } }
I20260927 02:33:41.253793 2017 raft_consensus.cc:374] T e8fe49295d774ba49452444a2fd07e2a P e62fb61a7e20421caccba2f459dd16fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:33:41.253821 2016 raft_consensus.cc:374] T e8fe49295d774ba49452444a2fd07e2a P e0e08aa3ca404cea99e4b5da45d4a9b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:33:41.253840 2017 raft_consensus.cc:729] T e8fe49295d774ba49452444a2fd07e2a P e62fb61a7e20421caccba2f459dd16fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e62fb61a7e20421caccba2f459dd16fb, State: Initialized, Role: FOLLOWER
I20260927 02:33:41.253868 2016 raft_consensus.cc:729] T e8fe49295d774ba49452444a2fd07e2a P e0e08aa3ca404cea99e4b5da45d4a9b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0e08aa3ca404cea99e4b5da45d4a9b0, State: Initialized, Role: FOLLOWER
I20260927 02:33:41.254230 2017 consensus_queue.cc:260] T e8fe49295d774ba49452444a2fd07e2a P e62fb61a7e20421caccba2f459dd16fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33553 } } peers { permanent_uuid: "e62fb61a7e20421caccba2f459dd16fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 } } peers { permanent_uuid: "3e020b8f631243e2a93860e0fb7bdff7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35549 } }
I20260927 02:33:41.254308 2016 consensus_queue.cc:260] T e8fe49295d774ba49452444a2fd07e2a P e0e08aa3ca404cea99e4b5da45d4a9b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33553 } } peers { permanent_uuid: "e62fb61a7e20421caccba2f459dd16fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 } } peers { permanent_uuid: "3e020b8f631243e2a93860e0fb7bdff7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35549 } }
I20260927 02:33:41.254781 1993 heartbeater.cc:503] Master 127.0.98.62:43451 was elected leader, sending a full tablet report...
I20260927 02:33:41.254717 2018 raft_consensus.cc:348] T e8fe49295d774ba49452444a2fd07e2a P 3e020b8f631243e2a93860e0fb7bdff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33553 } } peers { permanent_uuid: "e62fb61a7e20421caccba2f459dd16fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 } } peers { permanent_uuid: "3e020b8f631243e2a93860e0fb7bdff7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35549 } }
I20260927 02:33:41.254945 2018 raft_consensus.cc:374] T e8fe49295d774ba49452444a2fd07e2a P 3e020b8f631243e2a93860e0fb7bdff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:33:41.254990 2018 raft_consensus.cc:729] T e8fe49295d774ba49452444a2fd07e2a P 3e020b8f631243e2a93860e0fb7bdff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e020b8f631243e2a93860e0fb7bdff7, State: Initialized, Role: FOLLOWER
I20260927 02:33:41.255108 2017 ts_tablet_manager.cc:1428] T e8fe49295d774ba49452444a2fd07e2a P e62fb61a7e20421caccba2f459dd16fb: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260927 02:33:41.255187 2016 ts_tablet_manager.cc:1428] T e8fe49295d774ba49452444a2fd07e2a P e0e08aa3ca404cea99e4b5da45d4a9b0: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260927 02:33:41.255357 2018 consensus_queue.cc:260] T e8fe49295d774ba49452444a2fd07e2a P 3e020b8f631243e2a93860e0fb7bdff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33553 } } peers { permanent_uuid: "e62fb61a7e20421caccba2f459dd16fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 } } peers { permanent_uuid: "3e020b8f631243e2a93860e0fb7bdff7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35549 } }
I20260927 02:33:41.255478 1853 heartbeater.cc:503] Master 127.0.98.62:43451 was elected leader, sending a full tablet report...
I20260927 02:33:41.255962 1713 heartbeater.cc:503] Master 127.0.98.62:43451 was elected leader, sending a full tablet report...
I20260927 02:33:41.256171 2018 ts_tablet_manager.cc:1428] T e8fe49295d774ba49452444a2fd07e2a P 3e020b8f631243e2a93860e0fb7bdff7: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 02:33:41.263530 2023 raft_consensus.cc:482] T e8fe49295d774ba49452444a2fd07e2a P e0e08aa3ca404cea99e4b5da45d4a9b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:33:41.263669 2023 raft_consensus.cc:504] T e8fe49295d774ba49452444a2fd07e2a P e0e08aa3ca404cea99e4b5da45d4a9b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33553 } } peers { permanent_uuid: "e62fb61a7e20421caccba2f459dd16fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 } } peers { permanent_uuid: "3e020b8f631243e2a93860e0fb7bdff7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35549 } }
I20260927 02:33:41.264513 2023 leader_election.cc:288] T e8fe49295d774ba49452444a2fd07e2a P e0e08aa3ca404cea99e4b5da45d4a9b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e62fb61a7e20421caccba2f459dd16fb (127.0.98.3:44251), 3e020b8f631243e2a93860e0fb7bdff7 (127.0.98.1:35549)
I20260927 02:33:41.268442 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:41.264767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60197 (local address 127.0.98.3:44251)
0927 02:33:41.264872 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:41.264877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:41.264890 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:33:41.265063 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:41.265067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:41.265130 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:41.265203 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:33:41.265531 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:41.266152 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:41.267414 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:41.267986 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:41.268097 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:41.268156 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:41.268303 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 02:33:41.268354 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 02:33:41.268476 1746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:41.264718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44251 (local address 127.0.98.2:60197)
0927 02:33:41.264830 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:41.264845 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:41.264946 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:41.265314 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:41.265317 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:41.265336 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:33:41.265440 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:41.265447 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:41.266223 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:41.266226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:41.267302 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:41.267311 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:41.268078 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:41.268080 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:41.268187 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:41.268213 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 02:33:41.268280 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 02:33:41.268351 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":20}
I20260927 02:33:41.269091 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8fe49295d774ba49452444a2fd07e2a" candidate_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e62fb61a7e20421caccba2f459dd16fb" is_pre_election: true
I20260927 02:33:41.269167 2025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:41.264982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35549 (local address 127.0.98.2:45203)
0927 02:33:41.265233 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:41.265245 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:41.265328 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:41.265565 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:41.265567 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:41.265581 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:33:41.266043 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:41.266051 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:41.267003 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:41.267005 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:41.268045 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:41.268055 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:41.268804 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:41.268807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:41.268901 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:41.268920 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 02:33:41.269013 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 02:33:41.269062 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":1,"thread_start_us":114,"threads_started":1}
I20260927 02:33:41.269196 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:41.265062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45203 (local address 127.0.98.1:35549)
0927 02:33:41.265172 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:41.265176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:41.265303 (+ 127us) server_negotiation.cc:407] Connection header received
0927 02:33:41.265414 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:41.265417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:41.265457 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:41.265522 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:33:41.266150 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:41.266896 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:41.268146 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:41.268693 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:41.268784 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:41.268830 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:41.269027 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 02:33:41.269093 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 02:33:41.269366 1947 raft_consensus.cc:2454] T e8fe49295d774ba49452444a2fd07e2a P e62fb61a7e20421caccba2f459dd16fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0e08aa3ca404cea99e4b5da45d4a9b0 in term 0.
I20260927 02:33:41.269781 1741 leader_election.cc:302] T e8fe49295d774ba49452444a2fd07e2a P e0e08aa3ca404cea99e4b5da45d4a9b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0e08aa3ca404cea99e4b5da45d4a9b0, e62fb61a7e20421caccba2f459dd16fb; no voters:
I20260927 02:33:41.270020 2023 raft_consensus.cc:2781] T e8fe49295d774ba49452444a2fd07e2a P e0e08aa3ca404cea99e4b5da45d4a9b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:33:41.269987 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8fe49295d774ba49452444a2fd07e2a" candidate_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e020b8f631243e2a93860e0fb7bdff7" is_pre_election: true
I20260927 02:33:41.270092 2023 raft_consensus.cc:482] T e8fe49295d774ba49452444a2fd07e2a P e0e08aa3ca404cea99e4b5da45d4a9b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:33:41.270126 2023 raft_consensus.cc:3037] T e8fe49295d774ba49452444a2fd07e2a P e0e08aa3ca404cea99e4b5da45d4a9b0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:33:41.270234 1667 raft_consensus.cc:2454] T e8fe49295d774ba49452444a2fd07e2a P 3e020b8f631243e2a93860e0fb7bdff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0e08aa3ca404cea99e4b5da45d4a9b0 in term 0.
I20260927 02:33:41.271117 2023 raft_consensus.cc:504] T e8fe49295d774ba49452444a2fd07e2a P e0e08aa3ca404cea99e4b5da45d4a9b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33553 } } peers { permanent_uuid: "e62fb61a7e20421caccba2f459dd16fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 } } peers { permanent_uuid: "3e020b8f631243e2a93860e0fb7bdff7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35549 } }
I20260927 02:33:41.271399 2023 leader_election.cc:288] T e8fe49295d774ba49452444a2fd07e2a P e0e08aa3ca404cea99e4b5da45d4a9b0 [CANDIDATE]: Term 1 election: Requested vote from peers e62fb61a7e20421caccba2f459dd16fb (127.0.98.3:44251), 3e020b8f631243e2a93860e0fb7bdff7 (127.0.98.1:35549)
I20260927 02:33:41.271693 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8fe49295d774ba49452444a2fd07e2a" candidate_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e62fb61a7e20421caccba2f459dd16fb"
I20260927 02:33:41.271806 1947 raft_consensus.cc:3037] T e8fe49295d774ba49452444a2fd07e2a P e62fb61a7e20421caccba2f459dd16fb [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:33:41.271735 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8fe49295d774ba49452444a2fd07e2a" candidate_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e020b8f631243e2a93860e0fb7bdff7"
I20260927 02:33:41.271917 1667 raft_consensus.cc:3037] T e8fe49295d774ba49452444a2fd07e2a P 3e020b8f631243e2a93860e0fb7bdff7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:33:41.272725 1947 raft_consensus.cc:2454] T e8fe49295d774ba49452444a2fd07e2a P e62fb61a7e20421caccba2f459dd16fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0e08aa3ca404cea99e4b5da45d4a9b0 in term 1.
I20260927 02:33:41.272915 1667 raft_consensus.cc:2454] T e8fe49295d774ba49452444a2fd07e2a P 3e020b8f631243e2a93860e0fb7bdff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0e08aa3ca404cea99e4b5da45d4a9b0 in term 1.
I20260927 02:33:41.272991 1741 leader_election.cc:302] T e8fe49295d774ba49452444a2fd07e2a P e0e08aa3ca404cea99e4b5da45d4a9b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0e08aa3ca404cea99e4b5da45d4a9b0, e62fb61a7e20421caccba2f459dd16fb; no voters:
I20260927 02:33:41.273132 2023 raft_consensus.cc:2781] T e8fe49295d774ba49452444a2fd07e2a P e0e08aa3ca404cea99e4b5da45d4a9b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:33:41.273315 2023 raft_consensus.cc:686] T e8fe49295d774ba49452444a2fd07e2a P e0e08aa3ca404cea99e4b5da45d4a9b0 [term 1 LEADER]: Becoming Leader. State: Replica: e0e08aa3ca404cea99e4b5da45d4a9b0, State: Running, Role: LEADER
I20260927 02:33:41.273579 2023 consensus_queue.cc:237] T e8fe49295d774ba49452444a2fd07e2a P e0e08aa3ca404cea99e4b5da45d4a9b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33553 } } peers { permanent_uuid: "e62fb61a7e20421caccba2f459dd16fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 } } peers { permanent_uuid: "3e020b8f631243e2a93860e0fb7bdff7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35549 } }
I20260927 02:33:41.276289 1510 catalog_manager.cc:5495] T e8fe49295d774ba49452444a2fd07e2a P e0e08aa3ca404cea99e4b5da45d4a9b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0e08aa3ca404cea99e4b5da45d4a9b0 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e62fb61a7e20421caccba2f459dd16fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e020b8f631243e2a93860e0fb7bdff7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35549 } health_report { overall_health: UNKNOWN } } }
W20260927 02:33:41.285352 1714 tablet.cc:2369] T e8fe49295d774ba49452444a2fd07e2a P 3e020b8f631243e2a93860e0fb7bdff7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:33:41.299959 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:41.296587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37276 (local address 127.0.98.2:33553)
0927 02:33:41.296711 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:41.296715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:41.296728 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:33:41.296903 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:41.296906 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:41.296984 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:41.297065 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:33:41.297312 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:41.297793 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:41.298650 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:41.298868 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:41.298948 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:41.299254 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:33:41.299615 (+ 361us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:33:41.299709 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:41.299795 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 02:33:41.299842 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 02:33:41.305182 1667 raft_consensus.cc:1261] T e8fe49295d774ba49452444a2fd07e2a P 3e020b8f631243e2a93860e0fb7bdff7 [term 1 FOLLOWER]: Refusing update from remote peer e0e08aa3ca404cea99e4b5da45d4a9b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:33:41.305546 1947 raft_consensus.cc:1261] T e8fe49295d774ba49452444a2fd07e2a P e62fb61a7e20421caccba2f459dd16fb [term 1 FOLLOWER]: Refusing update from remote peer e0e08aa3ca404cea99e4b5da45d4a9b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:33:41.305763 2023 consensus_queue.cc:1035] T e8fe49295d774ba49452444a2fd07e2a P e0e08aa3ca404cea99e4b5da45d4a9b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e020b8f631243e2a93860e0fb7bdff7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:33:41.306068 2026 consensus_queue.cc:1035] T e8fe49295d774ba49452444a2fd07e2a P e0e08aa3ca404cea99e4b5da45d4a9b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e62fb61a7e20421caccba2f459dd16fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:33:41.312529 2031 mvcc.cc:205] Tried to move back new op lower bound from 7333791421660233728 to 7333791421538758656. Current Snapshot: MvccSnapshot[applied={T|T < 7333791421660233728}]
I20260927 02:33:41.323848 1510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:33:41.323984 1510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:33:41.326026 1510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0c09af2b5fc64aa3bf8204902b896b0a: Sending DeleteTablet for 3 replicas of tablet e8fe49295d774ba49452444a2fd07e2a
I20260927 02:33:41.326638 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet e8fe49295d774ba49452444a2fd07e2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:33:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37272
I20260927 02:33:41.327057 2038 tablet_replica.cc:330] T e8fe49295d774ba49452444a2fd07e2a P e0e08aa3ca404cea99e4b5da45d4a9b0: stopping tablet replica
I20260927 02:33:41.327039 1647 tablet_service.cc:1514] Processing DeleteTablet for tablet e8fe49295d774ba49452444a2fd07e2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:33:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48260
I20260927 02:33:41.327049 1927 tablet_service.cc:1514] Processing DeleteTablet for tablet e8fe49295d774ba49452444a2fd07e2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:33:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45054
I20260927 02:33:41.327426 2040 tablet_replica.cc:330] T e8fe49295d774ba49452444a2fd07e2a P e62fb61a7e20421caccba2f459dd16fb: stopping tablet replica
I20260927 02:33:41.327436 2038 raft_consensus.cc:2229] T e8fe49295d774ba49452444a2fd07e2a P e0e08aa3ca404cea99e4b5da45d4a9b0 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:33:41.327682 2040 raft_consensus.cc:2229] T e8fe49295d774ba49452444a2fd07e2a P e62fb61a7e20421caccba2f459dd16fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:33:41.327630 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45264:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 02:33:41.327945 2040 raft_consensus.cc:2258] T e8fe49295d774ba49452444a2fd07e2a P e62fb61a7e20421caccba2f459dd16fb [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 02:33:41.327961 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:33:41.328058 2038 raft_consensus.cc:2258] T e8fe49295d774ba49452444a2fd07e2a P e0e08aa3ca404cea99e4b5da45d4a9b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:33:41.329396 2040 ts_tablet_manager.cc:1905] T e8fe49295d774ba49452444a2fd07e2a P e62fb61a7e20421caccba2f459dd16fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:33:41.329396 2038 ts_tablet_manager.cc:1905] T e8fe49295d774ba49452444a2fd07e2a P e0e08aa3ca404cea99e4b5da45d4a9b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:33:41.331842 2040 ts_tablet_manager.cc:1918] T e8fe49295d774ba49452444a2fd07e2a P e62fb61a7e20421caccba2f459dd16fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:33:41.331928 2040 log.cc:1196] T e8fe49295d774ba49452444a2fd07e2a P e62fb61a7e20421caccba2f459dd16fb: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-2/wal/wals/e8fe49295d774ba49452444a2fd07e2a
I20260927 02:33:41.332221 2040 ts_tablet_manager.cc:1939] T e8fe49295d774ba49452444a2fd07e2a P e62fb61a7e20421caccba2f459dd16fb: Deleting consensus metadata
I20260927 02:33:41.332167 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 1494a4c70d954830bb9a1793fe3f5a50 (DEFAULT_TABLE table=test-table [id=249c233f52d64732aaf0698e4bd1dbed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:33:41.332405 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1494a4c70d954830bb9a1793fe3f5a50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:33:41.332569 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 1494a4c70d954830bb9a1793fe3f5a50 (DEFAULT_TABLE table=test-table [id=249c233f52d64732aaf0698e4bd1dbed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:33:41.332778 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1494a4c70d954830bb9a1793fe3f5a50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:33:41.332770 2039 tablet_replica.cc:330] T e8fe49295d774ba49452444a2fd07e2a P 3e020b8f631243e2a93860e0fb7bdff7: stopping tablet replica
I20260927 02:33:41.333024 2039 raft_consensus.cc:2229] T e8fe49295d774ba49452444a2fd07e2a P 3e020b8f631243e2a93860e0fb7bdff7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:33:41.333263 2039 raft_consensus.cc:2258] T e8fe49295d774ba49452444a2fd07e2a P 3e020b8f631243e2a93860e0fb7bdff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:33:41.334617 2039 ts_tablet_manager.cc:1905] T e8fe49295d774ba49452444a2fd07e2a P 3e020b8f631243e2a93860e0fb7bdff7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:33:41.334897 2016 tablet_bootstrap.cc:492] T 1494a4c70d954830bb9a1793fe3f5a50 P e0e08aa3ca404cea99e4b5da45d4a9b0: Bootstrap starting.
I20260927 02:33:41.335386 1927 tablet_service.cc:1467] Processing CreateTablet for tablet 1494a4c70d954830bb9a1793fe3f5a50 (DEFAULT_TABLE table=test-table [id=249c233f52d64732aaf0698e4bd1dbed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:33:41.335583 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1494a4c70d954830bb9a1793fe3f5a50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:33:41.335765 2016 tablet_bootstrap.cc:654] T 1494a4c70d954830bb9a1793fe3f5a50 P e0e08aa3ca404cea99e4b5da45d4a9b0: Neither blocks nor log segments found. Creating new log.
I20260927 02:33:41.336796 2018 tablet_bootstrap.cc:492] T 1494a4c70d954830bb9a1793fe3f5a50 P 3e020b8f631243e2a93860e0fb7bdff7: Bootstrap starting.
I20260927 02:33:41.337462 2039 ts_tablet_manager.cc:1918] T e8fe49295d774ba49452444a2fd07e2a P 3e020b8f631243e2a93860e0fb7bdff7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:33:41.337540 2039 log.cc:1196] T e8fe49295d774ba49452444a2fd07e2a P 3e020b8f631243e2a93860e0fb7bdff7: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-0/wal/wals/e8fe49295d774ba49452444a2fd07e2a
I20260927 02:33:41.337579 2017 tablet_bootstrap.cc:492] T 1494a4c70d954830bb9a1793fe3f5a50 P e62fb61a7e20421caccba2f459dd16fb: Bootstrap starting.
I20260927 02:33:41.337641 2018 tablet_bootstrap.cc:654] T 1494a4c70d954830bb9a1793fe3f5a50 P 3e020b8f631243e2a93860e0fb7bdff7: Neither blocks nor log segments found. Creating new log.
I20260927 02:33:41.337823 2039 ts_tablet_manager.cc:1939] T e8fe49295d774ba49452444a2fd07e2a P 3e020b8f631243e2a93860e0fb7bdff7: Deleting consensus metadata
I20260927 02:33:41.338394 2017 tablet_bootstrap.cc:654] T 1494a4c70d954830bb9a1793fe3f5a50 P e62fb61a7e20421caccba2f459dd16fb: Neither blocks nor log segments found. Creating new log.
I20260927 02:33:41.338863 1497 catalog_manager.cc:4841] TS 3e020b8f631243e2a93860e0fb7bdff7 (127.0.98.1:35549): tablet e8fe49295d774ba49452444a2fd07e2a (table test-table [id=df0ff8ccf7814024828d495b3d478c49]) successfully deleted
I20260927 02:33:41.339043 1497 catalog_manager.cc:4841] TS e62fb61a7e20421caccba2f459dd16fb (127.0.98.3:44251): tablet e8fe49295d774ba49452444a2fd07e2a (table test-table [id=df0ff8ccf7814024828d495b3d478c49]) successfully deleted
I20260927 02:33:41.339466 2038 ts_tablet_manager.cc:1918] T e8fe49295d774ba49452444a2fd07e2a P e0e08aa3ca404cea99e4b5da45d4a9b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:33:41.339483 2016 tablet_bootstrap.cc:492] T 1494a4c70d954830bb9a1793fe3f5a50 P e0e08aa3ca404cea99e4b5da45d4a9b0: No bootstrap required, opened a new log
I20260927 02:33:41.339622 2016 ts_tablet_manager.cc:1397] T 1494a4c70d954830bb9a1793fe3f5a50 P e0e08aa3ca404cea99e4b5da45d4a9b0: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260927 02:33:41.340093 2017 tablet_bootstrap.cc:492] T 1494a4c70d954830bb9a1793fe3f5a50 P e62fb61a7e20421caccba2f459dd16fb: No bootstrap required, opened a new log
I20260927 02:33:41.340026 2016 raft_consensus.cc:348] T 1494a4c70d954830bb9a1793fe3f5a50 P e0e08aa3ca404cea99e4b5da45d4a9b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e020b8f631243e2a93860e0fb7bdff7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35549 } } peers { permanent_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33553 } } peers { permanent_uuid: "e62fb61a7e20421caccba2f459dd16fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 } }
I20260927 02:33:41.340127 2016 raft_consensus.cc:374] T 1494a4c70d954830bb9a1793fe3f5a50 P e0e08aa3ca404cea99e4b5da45d4a9b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:33:41.340145 2017 ts_tablet_manager.cc:1397] T 1494a4c70d954830bb9a1793fe3f5a50 P e62fb61a7e20421caccba2f459dd16fb: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 02:33:41.340157 2016 raft_consensus.cc:729] T 1494a4c70d954830bb9a1793fe3f5a50 P e0e08aa3ca404cea99e4b5da45d4a9b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0e08aa3ca404cea99e4b5da45d4a9b0, State: Initialized, Role: FOLLOWER
I20260927 02:33:41.340260 2016 consensus_queue.cc:260] T 1494a4c70d954830bb9a1793fe3f5a50 P e0e08aa3ca404cea99e4b5da45d4a9b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e020b8f631243e2a93860e0fb7bdff7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35549 } } peers { permanent_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33553 } } peers { permanent_uuid: "e62fb61a7e20421caccba2f459dd16fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 } }
I20260927 02:33:41.340438 2017 raft_consensus.cc:348] T 1494a4c70d954830bb9a1793fe3f5a50 P e62fb61a7e20421caccba2f459dd16fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e020b8f631243e2a93860e0fb7bdff7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35549 } } peers { permanent_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33553 } } peers { permanent_uuid: "e62fb61a7e20421caccba2f459dd16fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 } }
I20260927 02:33:41.340483 2016 ts_tablet_manager.cc:1428] T 1494a4c70d954830bb9a1793fe3f5a50 P e0e08aa3ca404cea99e4b5da45d4a9b0: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 02:33:41.340504 2017 raft_consensus.cc:374] T 1494a4c70d954830bb9a1793fe3f5a50 P e62fb61a7e20421caccba2f459dd16fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:33:41.340523 2017 raft_consensus.cc:729] T 1494a4c70d954830bb9a1793fe3f5a50 P e62fb61a7e20421caccba2f459dd16fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e62fb61a7e20421caccba2f459dd16fb, State: Initialized, Role: FOLLOWER
I20260927 02:33:41.340622 2017 consensus_queue.cc:260] T 1494a4c70d954830bb9a1793fe3f5a50 P e62fb61a7e20421caccba2f459dd16fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e020b8f631243e2a93860e0fb7bdff7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35549 } } peers { permanent_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33553 } } peers { permanent_uuid: "e62fb61a7e20421caccba2f459dd16fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 } }
I20260927 02:33:41.340838 2017 ts_tablet_manager.cc:1428] T 1494a4c70d954830bb9a1793fe3f5a50 P e62fb61a7e20421caccba2f459dd16fb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:33:41.340847 2038 log.cc:1196] T e8fe49295d774ba49452444a2fd07e2a P e0e08aa3ca404cea99e4b5da45d4a9b0: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-1/wal/wals/e8fe49295d774ba49452444a2fd07e2a
I20260927 02:33:41.341229 2038 ts_tablet_manager.cc:1939] T e8fe49295d774ba49452444a2fd07e2a P e0e08aa3ca404cea99e4b5da45d4a9b0: Deleting consensus metadata
I20260927 02:33:41.342147 1494 catalog_manager.cc:4841] TS e0e08aa3ca404cea99e4b5da45d4a9b0 (127.0.98.2:33553): tablet e8fe49295d774ba49452444a2fd07e2a (table test-table [id=df0ff8ccf7814024828d495b3d478c49]) successfully deleted
I20260927 02:33:41.342932 2018 tablet_bootstrap.cc:492] T 1494a4c70d954830bb9a1793fe3f5a50 P 3e020b8f631243e2a93860e0fb7bdff7: No bootstrap required, opened a new log
I20260927 02:33:41.343008 2018 ts_tablet_manager.cc:1397] T 1494a4c70d954830bb9a1793fe3f5a50 P 3e020b8f631243e2a93860e0fb7bdff7: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260927 02:33:41.343386 2018 raft_consensus.cc:348] T 1494a4c70d954830bb9a1793fe3f5a50 P 3e020b8f631243e2a93860e0fb7bdff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e020b8f631243e2a93860e0fb7bdff7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35549 } } peers { permanent_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33553 } } peers { permanent_uuid: "e62fb61a7e20421caccba2f459dd16fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 } }
I20260927 02:33:41.343490 2018 raft_consensus.cc:374] T 1494a4c70d954830bb9a1793fe3f5a50 P 3e020b8f631243e2a93860e0fb7bdff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:33:41.343519 2018 raft_consensus.cc:729] T 1494a4c70d954830bb9a1793fe3f5a50 P 3e020b8f631243e2a93860e0fb7bdff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e020b8f631243e2a93860e0fb7bdff7, State: Initialized, Role: FOLLOWER
I20260927 02:33:41.343627 2018 consensus_queue.cc:260] T 1494a4c70d954830bb9a1793fe3f5a50 P 3e020b8f631243e2a93860e0fb7bdff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e020b8f631243e2a93860e0fb7bdff7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35549 } } peers { permanent_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33553 } } peers { permanent_uuid: "e62fb61a7e20421caccba2f459dd16fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 } }
I20260927 02:33:41.343855 2018 ts_tablet_manager.cc:1428] T 1494a4c70d954830bb9a1793fe3f5a50 P 3e020b8f631243e2a93860e0fb7bdff7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 02:33:41.352864 2026 raft_consensus.cc:482] T 1494a4c70d954830bb9a1793fe3f5a50 P e0e08aa3ca404cea99e4b5da45d4a9b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:33:41.352969 2026 raft_consensus.cc:504] T 1494a4c70d954830bb9a1793fe3f5a50 P e0e08aa3ca404cea99e4b5da45d4a9b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e020b8f631243e2a93860e0fb7bdff7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35549 } } peers { permanent_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33553 } } peers { permanent_uuid: "e62fb61a7e20421caccba2f459dd16fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 } }
I20260927 02:33:41.353281 2026 leader_election.cc:288] T 1494a4c70d954830bb9a1793fe3f5a50 P e0e08aa3ca404cea99e4b5da45d4a9b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e020b8f631243e2a93860e0fb7bdff7 (127.0.98.1:35549), e62fb61a7e20421caccba2f459dd16fb (127.0.98.3:44251)
I20260927 02:33:41.353551 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1494a4c70d954830bb9a1793fe3f5a50" candidate_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e020b8f631243e2a93860e0fb7bdff7" is_pre_election: true
I20260927 02:33:41.353675 1667 raft_consensus.cc:2454] T 1494a4c70d954830bb9a1793fe3f5a50 P 3e020b8f631243e2a93860e0fb7bdff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0e08aa3ca404cea99e4b5da45d4a9b0 in term 0.
I20260927 02:33:41.353650 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1494a4c70d954830bb9a1793fe3f5a50" candidate_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e62fb61a7e20421caccba2f459dd16fb" is_pre_election: true
I20260927 02:33:41.353765 1947 raft_consensus.cc:2454] T 1494a4c70d954830bb9a1793fe3f5a50 P e62fb61a7e20421caccba2f459dd16fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0e08aa3ca404cea99e4b5da45d4a9b0 in term 0.
I20260927 02:33:41.353941 1741 leader_election.cc:302] T 1494a4c70d954830bb9a1793fe3f5a50 P e0e08aa3ca404cea99e4b5da45d4a9b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e020b8f631243e2a93860e0fb7bdff7, e0e08aa3ca404cea99e4b5da45d4a9b0; no voters:
I20260927 02:33:41.354120 2026 raft_consensus.cc:2781] T 1494a4c70d954830bb9a1793fe3f5a50 P e0e08aa3ca404cea99e4b5da45d4a9b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:33:41.354180 2026 raft_consensus.cc:482] T 1494a4c70d954830bb9a1793fe3f5a50 P e0e08aa3ca404cea99e4b5da45d4a9b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:33:41.354236 2026 raft_consensus.cc:3037] T 1494a4c70d954830bb9a1793fe3f5a50 P e0e08aa3ca404cea99e4b5da45d4a9b0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:33:41.355280 2026 raft_consensus.cc:504] T 1494a4c70d954830bb9a1793fe3f5a50 P e0e08aa3ca404cea99e4b5da45d4a9b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e020b8f631243e2a93860e0fb7bdff7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35549 } } peers { permanent_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33553 } } peers { permanent_uuid: "e62fb61a7e20421caccba2f459dd16fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 } }
I20260927 02:33:41.355587 2026 leader_election.cc:288] T 1494a4c70d954830bb9a1793fe3f5a50 P e0e08aa3ca404cea99e4b5da45d4a9b0 [CANDIDATE]: Term 1 election: Requested vote from peers 3e020b8f631243e2a93860e0fb7bdff7 (127.0.98.1:35549), e62fb61a7e20421caccba2f459dd16fb (127.0.98.3:44251)
I20260927 02:33:41.355852 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1494a4c70d954830bb9a1793fe3f5a50" candidate_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e020b8f631243e2a93860e0fb7bdff7"
I20260927 02:33:41.355898 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1494a4c70d954830bb9a1793fe3f5a50" candidate_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e62fb61a7e20421caccba2f459dd16fb"
I20260927 02:33:41.355957 1667 raft_consensus.cc:3037] T 1494a4c70d954830bb9a1793fe3f5a50 P 3e020b8f631243e2a93860e0fb7bdff7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:33:41.355983 1947 raft_consensus.cc:3037] T 1494a4c70d954830bb9a1793fe3f5a50 P e62fb61a7e20421caccba2f459dd16fb [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:33:41.356887 1667 raft_consensus.cc:2454] T 1494a4c70d954830bb9a1793fe3f5a50 P 3e020b8f631243e2a93860e0fb7bdff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0e08aa3ca404cea99e4b5da45d4a9b0 in term 1.
I20260927 02:33:41.356887 1947 raft_consensus.cc:2454] T 1494a4c70d954830bb9a1793fe3f5a50 P e62fb61a7e20421caccba2f459dd16fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0e08aa3ca404cea99e4b5da45d4a9b0 in term 1.
I20260927 02:33:41.357149 1741 leader_election.cc:302] T 1494a4c70d954830bb9a1793fe3f5a50 P e0e08aa3ca404cea99e4b5da45d4a9b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e020b8f631243e2a93860e0fb7bdff7, e0e08aa3ca404cea99e4b5da45d4a9b0; no voters:
I20260927 02:33:41.357302 2026 raft_consensus.cc:2781] T 1494a4c70d954830bb9a1793fe3f5a50 P e0e08aa3ca404cea99e4b5da45d4a9b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:33:41.357394 2026 raft_consensus.cc:686] T 1494a4c70d954830bb9a1793fe3f5a50 P e0e08aa3ca404cea99e4b5da45d4a9b0 [term 1 LEADER]: Becoming Leader. State: Replica: e0e08aa3ca404cea99e4b5da45d4a9b0, State: Running, Role: LEADER
I20260927 02:33:41.357511 2026 consensus_queue.cc:237] T 1494a4c70d954830bb9a1793fe3f5a50 P e0e08aa3ca404cea99e4b5da45d4a9b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e020b8f631243e2a93860e0fb7bdff7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35549 } } peers { permanent_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33553 } } peers { permanent_uuid: "e62fb61a7e20421caccba2f459dd16fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 } }
I20260927 02:33:41.358878 1510 catalog_manager.cc:5495] T 1494a4c70d954830bb9a1793fe3f5a50 P e0e08aa3ca404cea99e4b5da45d4a9b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0e08aa3ca404cea99e4b5da45d4a9b0 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e020b8f631243e2a93860e0fb7bdff7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e62fb61a7e20421caccba2f459dd16fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 } health_report { overall_health: UNKNOWN } } }
Sep 27 02:33:41 dist-test-slave-grlk krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:33:41.394905 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:41.386597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45270 (local address 127.0.98.62:43451)
0927 02:33:41.386703 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:41.386707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:41.386903 (+ 196us) server_negotiation.cc:407] Connection header received
0927 02:33:41.387025 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:41.387028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:41.387075 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:41.387137 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:41.387771 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:41.388282 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:41.389231 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:41.389428 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:41.389547 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:41.391768 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:41.391771 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:41.391773 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:41.391793 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:41.393137 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:41.393678 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:41.393681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:41.393682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:41.393762 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:41.394022 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:41.394024 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:41.394025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:41.394181 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:41.394249 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:41.394559 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 02:33:41.394716 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":85,"server-negotiator.queue_time_us":36}
I20260927 02:33:41.394922 2050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:41.386558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43451 (local address 127.0.0.1:45270)
0927 02:33:41.386822 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:41.386834 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:41.386942 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:41.387168 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:41.387170 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:41.387573 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:33:41.387660 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:41.387667 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:41.388374 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:41.388378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:41.389134 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:41.389143 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:41.389521 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:41.389525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:41.389665 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:41.390220 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:33:41.390232 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:33:41.391639 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:33:41.393231 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:41.393242 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:41.393249 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:41.393572 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:41.393834 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:41.393837 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:41.393839 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:41.393934 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:41.394265 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:33:41.394272 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:33:41.394400 (+ 128us) client_negotiation.cc:769] Sending connection context
0927 02:33:41.394518 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 02:33:41.394669 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":30,"thread_start_us":112,"threads_started":1}
I20260927 02:33:41.398645 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45270:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 02:33:41.399406 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:33:41.403900 1647 tablet_service.cc:1467] Processing CreateTablet for tablet a6cf091627ce409892e08b3a27a65c43 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=765de4f511cd4981a2a630797ae90d4e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:33:41.404031 1787 tablet_service.cc:1467] Processing CreateTablet for tablet a6cf091627ce409892e08b3a27a65c43 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=765de4f511cd4981a2a630797ae90d4e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:33:41.404031 1927 tablet_service.cc:1467] Processing CreateTablet for tablet a6cf091627ce409892e08b3a27a65c43 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=765de4f511cd4981a2a630797ae90d4e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:33:41.404277 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6cf091627ce409892e08b3a27a65c43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:33:41.404359 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6cf091627ce409892e08b3a27a65c43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:33:41.404359 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6cf091627ce409892e08b3a27a65c43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:33:41.406950 2016 tablet_bootstrap.cc:492] T a6cf091627ce409892e08b3a27a65c43 P e0e08aa3ca404cea99e4b5da45d4a9b0: Bootstrap starting.
I20260927 02:33:41.407049 2017 tablet_bootstrap.cc:492] T a6cf091627ce409892e08b3a27a65c43 P e62fb61a7e20421caccba2f459dd16fb: Bootstrap starting.
I20260927 02:33:41.407065 2018 tablet_bootstrap.cc:492] T a6cf091627ce409892e08b3a27a65c43 P 3e020b8f631243e2a93860e0fb7bdff7: Bootstrap starting.
I20260927 02:33:41.407861 2016 tablet_bootstrap.cc:654] T a6cf091627ce409892e08b3a27a65c43 P e0e08aa3ca404cea99e4b5da45d4a9b0: Neither blocks nor log segments found. Creating new log.
I20260927 02:33:41.407872 2017 tablet_bootstrap.cc:654] T a6cf091627ce409892e08b3a27a65c43 P e62fb61a7e20421caccba2f459dd16fb: Neither blocks nor log segments found. Creating new log.
I20260927 02:33:41.407992 2018 tablet_bootstrap.cc:654] T a6cf091627ce409892e08b3a27a65c43 P 3e020b8f631243e2a93860e0fb7bdff7: Neither blocks nor log segments found. Creating new log.
I20260927 02:33:41.409734 2017 tablet_bootstrap.cc:492] T a6cf091627ce409892e08b3a27a65c43 P e62fb61a7e20421caccba2f459dd16fb: No bootstrap required, opened a new log
I20260927 02:33:41.409811 2018 tablet_bootstrap.cc:492] T a6cf091627ce409892e08b3a27a65c43 P 3e020b8f631243e2a93860e0fb7bdff7: No bootstrap required, opened a new log
I20260927 02:33:41.409811 2017 ts_tablet_manager.cc:1397] T a6cf091627ce409892e08b3a27a65c43 P e62fb61a7e20421caccba2f459dd16fb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 02:33:41.409878 2018 ts_tablet_manager.cc:1397] T a6cf091627ce409892e08b3a27a65c43 P 3e020b8f631243e2a93860e0fb7bdff7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 02:33:41.409907 2016 tablet_bootstrap.cc:492] T a6cf091627ce409892e08b3a27a65c43 P e0e08aa3ca404cea99e4b5da45d4a9b0: No bootstrap required, opened a new log
I20260927 02:33:41.409972 2016 ts_tablet_manager.cc:1397] T a6cf091627ce409892e08b3a27a65c43 P e0e08aa3ca404cea99e4b5da45d4a9b0: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260927 02:33:41.410209 2017 raft_consensus.cc:348] T a6cf091627ce409892e08b3a27a65c43 P e62fb61a7e20421caccba2f459dd16fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e020b8f631243e2a93860e0fb7bdff7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35549 } } peers { permanent_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33553 } } peers { permanent_uuid: "e62fb61a7e20421caccba2f459dd16fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 } }
I20260927 02:33:41.410306 2017 raft_consensus.cc:374] T a6cf091627ce409892e08b3a27a65c43 P e62fb61a7e20421caccba2f459dd16fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:33:41.410254 2018 raft_consensus.cc:348] T a6cf091627ce409892e08b3a27a65c43 P 3e020b8f631243e2a93860e0fb7bdff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e020b8f631243e2a93860e0fb7bdff7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35549 } } peers { permanent_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33553 } } peers { permanent_uuid: "e62fb61a7e20421caccba2f459dd16fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 } }
I20260927 02:33:41.410337 2017 raft_consensus.cc:729] T a6cf091627ce409892e08b3a27a65c43 P e62fb61a7e20421caccba2f459dd16fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e62fb61a7e20421caccba2f459dd16fb, State: Initialized, Role: FOLLOWER
I20260927 02:33:41.410346 2018 raft_consensus.cc:374] T a6cf091627ce409892e08b3a27a65c43 P 3e020b8f631243e2a93860e0fb7bdff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:33:41.410373 2018 raft_consensus.cc:729] T a6cf091627ce409892e08b3a27a65c43 P 3e020b8f631243e2a93860e0fb7bdff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e020b8f631243e2a93860e0fb7bdff7, State: Initialized, Role: FOLLOWER
I20260927 02:33:41.410339 2016 raft_consensus.cc:348] T a6cf091627ce409892e08b3a27a65c43 P e0e08aa3ca404cea99e4b5da45d4a9b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e020b8f631243e2a93860e0fb7bdff7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35549 } } peers { permanent_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33553 } } peers { permanent_uuid: "e62fb61a7e20421caccba2f459dd16fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 } }
I20260927 02:33:41.410424 2016 raft_consensus.cc:374] T a6cf091627ce409892e08b3a27a65c43 P e0e08aa3ca404cea99e4b5da45d4a9b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:33:41.410454 2016 raft_consensus.cc:729] T a6cf091627ce409892e08b3a27a65c43 P e0e08aa3ca404cea99e4b5da45d4a9b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0e08aa3ca404cea99e4b5da45d4a9b0, State: Initialized, Role: FOLLOWER
I20260927 02:33:41.410456 2017 consensus_queue.cc:260] T a6cf091627ce409892e08b3a27a65c43 P e62fb61a7e20421caccba2f459dd16fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e020b8f631243e2a93860e0fb7bdff7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35549 } } peers { permanent_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33553 } } peers { permanent_uuid: "e62fb61a7e20421caccba2f459dd16fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 } }
I20260927 02:33:41.410476 2018 consensus_queue.cc:260] T a6cf091627ce409892e08b3a27a65c43 P 3e020b8f631243e2a93860e0fb7bdff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e020b8f631243e2a93860e0fb7bdff7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35549 } } peers { permanent_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33553 } } peers { permanent_uuid: "e62fb61a7e20421caccba2f459dd16fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 } }
I20260927 02:33:41.410555 2016 consensus_queue.cc:260] T a6cf091627ce409892e08b3a27a65c43 P e0e08aa3ca404cea99e4b5da45d4a9b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e020b8f631243e2a93860e0fb7bdff7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35549 } } peers { permanent_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33553 } } peers { permanent_uuid: "e62fb61a7e20421caccba2f459dd16fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 } }
I20260927 02:33:41.410697 2018 ts_tablet_manager.cc:1428] T a6cf091627ce409892e08b3a27a65c43 P 3e020b8f631243e2a93860e0fb7bdff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:33:41.410698 2017 ts_tablet_manager.cc:1428] T a6cf091627ce409892e08b3a27a65c43 P e62fb61a7e20421caccba2f459dd16fb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:33:41.410772 2016 ts_tablet_manager.cc:1428] T a6cf091627ce409892e08b3a27a65c43 P e0e08aa3ca404cea99e4b5da45d4a9b0: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 02:33:41.410928 2022 tablet_replica.cc:439] T a6cf091627ce409892e08b3a27a65c43 P e62fb61a7e20421caccba2f459dd16fb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e020b8f631243e2a93860e0fb7bdff7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35549 } } peers { permanent_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33553 } } peers { permanent_uuid: "e62fb61a7e20421caccba2f459dd16fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 } } }
I20260927 02:33:41.410961 2024 tablet_replica.cc:439] T a6cf091627ce409892e08b3a27a65c43 P 3e020b8f631243e2a93860e0fb7bdff7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e020b8f631243e2a93860e0fb7bdff7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35549 } } peers { permanent_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33553 } } peers { permanent_uuid: "e62fb61a7e20421caccba2f459dd16fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 } } }
I20260927 02:33:41.411067 2022 tablet_replica.cc:442] T a6cf091627ce409892e08b3a27a65c43 P e62fb61a7e20421caccba2f459dd16fb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:33:41.411065 2024 tablet_replica.cc:442] T a6cf091627ce409892e08b3a27a65c43 P 3e020b8f631243e2a93860e0fb7bdff7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:33:41.411226 2026 tablet_replica.cc:439] T a6cf091627ce409892e08b3a27a65c43 P e0e08aa3ca404cea99e4b5da45d4a9b0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e020b8f631243e2a93860e0fb7bdff7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35549 } } peers { permanent_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33553 } } peers { permanent_uuid: "e62fb61a7e20421caccba2f459dd16fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 } } }
I20260927 02:33:41.411326 2026 tablet_replica.cc:442] T a6cf091627ce409892e08b3a27a65c43 P e0e08aa3ca404cea99e4b5da45d4a9b0: This TxnStatusTablet replica's current role is: FOLLOWER
W20260927 02:33:41.437290 1994 tablet.cc:2369] T 1494a4c70d954830bb9a1793fe3f5a50 P e62fb61a7e20421caccba2f459dd16fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:33:41.465271 2026 raft_consensus.cc:482] T a6cf091627ce409892e08b3a27a65c43 P e0e08aa3ca404cea99e4b5da45d4a9b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:33:41.465427 2026 raft_consensus.cc:504] T a6cf091627ce409892e08b3a27a65c43 P e0e08aa3ca404cea99e4b5da45d4a9b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e020b8f631243e2a93860e0fb7bdff7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35549 } } peers { permanent_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33553 } } peers { permanent_uuid: "e62fb61a7e20421caccba2f459dd16fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 } }
I20260927 02:33:41.465842 2026 leader_election.cc:288] T a6cf091627ce409892e08b3a27a65c43 P e0e08aa3ca404cea99e4b5da45d4a9b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e020b8f631243e2a93860e0fb7bdff7 (127.0.98.1:35549), e62fb61a7e20421caccba2f459dd16fb (127.0.98.3:44251)
I20260927 02:33:41.466245 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6cf091627ce409892e08b3a27a65c43" candidate_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e62fb61a7e20421caccba2f459dd16fb" is_pre_election: true
I20260927 02:33:41.466250 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6cf091627ce409892e08b3a27a65c43" candidate_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e020b8f631243e2a93860e0fb7bdff7" is_pre_election: true
I20260927 02:33:41.466395 1947 raft_consensus.cc:2454] T a6cf091627ce409892e08b3a27a65c43 P e62fb61a7e20421caccba2f459dd16fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0e08aa3ca404cea99e4b5da45d4a9b0 in term 0.
I20260927 02:33:41.466405 1667 raft_consensus.cc:2454] T a6cf091627ce409892e08b3a27a65c43 P 3e020b8f631243e2a93860e0fb7bdff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0e08aa3ca404cea99e4b5da45d4a9b0 in term 0.
I20260927 02:33:41.466660 1741 leader_election.cc:302] T a6cf091627ce409892e08b3a27a65c43 P e0e08aa3ca404cea99e4b5da45d4a9b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0e08aa3ca404cea99e4b5da45d4a9b0, e62fb61a7e20421caccba2f459dd16fb; no voters:
I20260927 02:33:41.466948 2026 raft_consensus.cc:2781] T a6cf091627ce409892e08b3a27a65c43 P e0e08aa3ca404cea99e4b5da45d4a9b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:33:41.467002 2026 raft_consensus.cc:482] T a6cf091627ce409892e08b3a27a65c43 P e0e08aa3ca404cea99e4b5da45d4a9b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:33:41.467044 2026 raft_consensus.cc:3037] T a6cf091627ce409892e08b3a27a65c43 P e0e08aa3ca404cea99e4b5da45d4a9b0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:33:41.468029 2026 raft_consensus.cc:504] T a6cf091627ce409892e08b3a27a65c43 P e0e08aa3ca404cea99e4b5da45d4a9b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e020b8f631243e2a93860e0fb7bdff7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35549 } } peers { permanent_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33553 } } peers { permanent_uuid: "e62fb61a7e20421caccba2f459dd16fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 } }
I20260927 02:33:41.468328 2026 leader_election.cc:288] T a6cf091627ce409892e08b3a27a65c43 P e0e08aa3ca404cea99e4b5da45d4a9b0 [CANDIDATE]: Term 1 election: Requested vote from peers 3e020b8f631243e2a93860e0fb7bdff7 (127.0.98.1:35549), e62fb61a7e20421caccba2f459dd16fb (127.0.98.3:44251)
I20260927 02:33:41.468636 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6cf091627ce409892e08b3a27a65c43" candidate_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e020b8f631243e2a93860e0fb7bdff7"
I20260927 02:33:41.468755 1667 raft_consensus.cc:3037] T a6cf091627ce409892e08b3a27a65c43 P 3e020b8f631243e2a93860e0fb7bdff7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:33:41.468722 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6cf091627ce409892e08b3a27a65c43" candidate_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e62fb61a7e20421caccba2f459dd16fb"
I20260927 02:33:41.469043 1947 raft_consensus.cc:3037] T a6cf091627ce409892e08b3a27a65c43 P e62fb61a7e20421caccba2f459dd16fb [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:33:41.469811 1667 raft_consensus.cc:2454] T a6cf091627ce409892e08b3a27a65c43 P 3e020b8f631243e2a93860e0fb7bdff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0e08aa3ca404cea99e4b5da45d4a9b0 in term 1.
I20260927 02:33:41.470060 1947 raft_consensus.cc:2454] T a6cf091627ce409892e08b3a27a65c43 P e62fb61a7e20421caccba2f459dd16fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0e08aa3ca404cea99e4b5da45d4a9b0 in term 1.
I20260927 02:33:41.470337 1741 leader_election.cc:302] T a6cf091627ce409892e08b3a27a65c43 P e0e08aa3ca404cea99e4b5da45d4a9b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0e08aa3ca404cea99e4b5da45d4a9b0, e62fb61a7e20421caccba2f459dd16fb; no voters:
I20260927 02:33:41.470513 2026 raft_consensus.cc:2781] T a6cf091627ce409892e08b3a27a65c43 P e0e08aa3ca404cea99e4b5da45d4a9b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:33:41.470602 2026 raft_consensus.cc:686] T a6cf091627ce409892e08b3a27a65c43 P e0e08aa3ca404cea99e4b5da45d4a9b0 [term 1 LEADER]: Becoming Leader. State: Replica: e0e08aa3ca404cea99e4b5da45d4a9b0, State: Running, Role: LEADER
I20260927 02:33:41.470721 2026 consensus_queue.cc:237] T a6cf091627ce409892e08b3a27a65c43 P e0e08aa3ca404cea99e4b5da45d4a9b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e020b8f631243e2a93860e0fb7bdff7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35549 } } peers { permanent_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33553 } } peers { permanent_uuid: "e62fb61a7e20421caccba2f459dd16fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 } }
I20260927 02:33:41.471335 2023 tablet_replica.cc:439] T a6cf091627ce409892e08b3a27a65c43 P e0e08aa3ca404cea99e4b5da45d4a9b0: TxnStatusTablet state changed. Reason: New leader e0e08aa3ca404cea99e4b5da45d4a9b0. Latest consensus state: current_term: 1 leader_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e020b8f631243e2a93860e0fb7bdff7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35549 } } peers { permanent_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33553 } } peers { permanent_uuid: "e62fb61a7e20421caccba2f459dd16fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 } } }
I20260927 02:33:41.471431 2023 tablet_replica.cc:442] T a6cf091627ce409892e08b3a27a65c43 P e0e08aa3ca404cea99e4b5da45d4a9b0: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:33:41.471763 2055 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 02:33:41.471907 2055 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 02:33:41.472280 1508 catalog_manager.cc:5495] T a6cf091627ce409892e08b3a27a65c43 P e0e08aa3ca404cea99e4b5da45d4a9b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0e08aa3ca404cea99e4b5da45d4a9b0 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e020b8f631243e2a93860e0fb7bdff7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e62fb61a7e20421caccba2f459dd16fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 } health_report { overall_health: UNKNOWN } } }
W20260927 02:33:41.489635 1854 tablet.cc:2369] T 1494a4c70d954830bb9a1793fe3f5a50 P e0e08aa3ca404cea99e4b5da45d4a9b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:33:41.494596 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:41.491284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37282 (local address 127.0.98.2:33553)
0927 02:33:41.491399 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:41.491403 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:41.491414 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:33:41.491543 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:41.491545 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:41.491592 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:41.491648 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:33:41.491888 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:41.492364 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:41.493552 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:41.493825 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:41.493922 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:41.494200 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:33:41.494275 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:33:41.494335 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:41.494428 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 02:33:41.494468 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 02:33:41.494634 2050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:41.491183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33553 (local address 127.0.0.1:37282)
0927 02:33:41.491335 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:41.491351 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:41.491447 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:41.491684 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:41.491687 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:41.491694 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:33:41.491798 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:41.491804 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:41.492449 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:41.492453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:41.493445 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:41.493455 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:41.493920 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:41.493924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:41.494022 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:41.494072 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:33:41.494348 (+ 276us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:33:41.494358 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 02:33:41.494419 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 02:33:41.494499 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 02:33:41.497074 1947 raft_consensus.cc:1261] T a6cf091627ce409892e08b3a27a65c43 P e62fb61a7e20421caccba2f459dd16fb [term 1 FOLLOWER]: Refusing update from remote peer e0e08aa3ca404cea99e4b5da45d4a9b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:33:41.497084 1667 raft_consensus.cc:1261] T a6cf091627ce409892e08b3a27a65c43 P 3e020b8f631243e2a93860e0fb7bdff7 [term 1 FOLLOWER]: Refusing update from remote peer e0e08aa3ca404cea99e4b5da45d4a9b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:33:41.497439 2026 consensus_queue.cc:1035] T a6cf091627ce409892e08b3a27a65c43 P e0e08aa3ca404cea99e4b5da45d4a9b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e62fb61a7e20421caccba2f459dd16fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:33:41.497578 2023 consensus_queue.cc:1035] T a6cf091627ce409892e08b3a27a65c43 P e0e08aa3ca404cea99e4b5da45d4a9b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e020b8f631243e2a93860e0fb7bdff7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:33:41.498777 2024 tablet_replica.cc:439] T a6cf091627ce409892e08b3a27a65c43 P 3e020b8f631243e2a93860e0fb7bdff7: TxnStatusTablet state changed. Reason: New leader e0e08aa3ca404cea99e4b5da45d4a9b0. Latest consensus state: current_term: 1 leader_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e020b8f631243e2a93860e0fb7bdff7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35549 } } peers { permanent_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33553 } } peers { permanent_uuid: "e62fb61a7e20421caccba2f459dd16fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 } } }
I20260927 02:33:41.498894 2024 tablet_replica.cc:442] T a6cf091627ce409892e08b3a27a65c43 P 3e020b8f631243e2a93860e0fb7bdff7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:33:41.498836 2022 tablet_replica.cc:439] T a6cf091627ce409892e08b3a27a65c43 P e62fb61a7e20421caccba2f459dd16fb: TxnStatusTablet state changed. Reason: New leader e0e08aa3ca404cea99e4b5da45d4a9b0. Latest consensus state: current_term: 1 leader_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e020b8f631243e2a93860e0fb7bdff7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35549 } } peers { permanent_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33553 } } peers { permanent_uuid: "e62fb61a7e20421caccba2f459dd16fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 } } }
I20260927 02:33:41.498925 2022 tablet_replica.cc:442] T a6cf091627ce409892e08b3a27a65c43 P e62fb61a7e20421caccba2f459dd16fb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:33:41.500846 2029 mvcc.cc:205] Tried to move back new op lower bound from 7333791422448111616 to 7333791422345437184. Current Snapshot: MvccSnapshot[applied={T|T < 7333791422448111616}]
I20260927 02:33:41.501081 2026 tablet_replica.cc:439] T a6cf091627ce409892e08b3a27a65c43 P e0e08aa3ca404cea99e4b5da45d4a9b0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e020b8f631243e2a93860e0fb7bdff7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35549 } } peers { permanent_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33553 } } peers { permanent_uuid: "e62fb61a7e20421caccba2f459dd16fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 } } }
I20260927 02:33:41.501173 2026 tablet_replica.cc:442] T a6cf091627ce409892e08b3a27a65c43 P e0e08aa3ca404cea99e4b5da45d4a9b0: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:33:41.501273 2026 tablet_replica.cc:439] T a6cf091627ce409892e08b3a27a65c43 P e0e08aa3ca404cea99e4b5da45d4a9b0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e020b8f631243e2a93860e0fb7bdff7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35549 } } peers { permanent_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33553 } } peers { permanent_uuid: "e62fb61a7e20421caccba2f459dd16fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 } } }
I20260927 02:33:41.501335 2026 tablet_replica.cc:442] T a6cf091627ce409892e08b3a27a65c43 P e0e08aa3ca404cea99e4b5da45d4a9b0: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:33:41.501528 2022 tablet_replica.cc:439] T a6cf091627ce409892e08b3a27a65c43 P e62fb61a7e20421caccba2f459dd16fb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e020b8f631243e2a93860e0fb7bdff7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35549 } } peers { permanent_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33553 } } peers { permanent_uuid: "e62fb61a7e20421caccba2f459dd16fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 } } }
I20260927 02:33:41.501631 2022 tablet_replica.cc:442] T a6cf091627ce409892e08b3a27a65c43 P e62fb61a7e20421caccba2f459dd16fb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:33:41.503144 2030 mvcc.cc:205] Tried to move back new op lower bound from 7333791422448111616 to 7333791422345437184. Current Snapshot: MvccSnapshot[applied={T|T < 7333791422448111616}]
I20260927 02:33:41.503300 2024 tablet_replica.cc:439] T a6cf091627ce409892e08b3a27a65c43 P 3e020b8f631243e2a93860e0fb7bdff7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e020b8f631243e2a93860e0fb7bdff7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35549 } } peers { permanent_uuid: "e0e08aa3ca404cea99e4b5da45d4a9b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33553 } } peers { permanent_uuid: "e62fb61a7e20421caccba2f459dd16fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 } } }
I20260927 02:33:41.503392 2024 tablet_replica.cc:442] T a6cf091627ce409892e08b3a27a65c43 P 3e020b8f631243e2a93860e0fb7bdff7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:33:41.514552 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:41.511507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57507 (local address 127.0.98.2:33553)
0927 02:33:41.511606 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:41.511610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:41.511786 (+ 176us) server_negotiation.cc:407] Connection header received
0927 02:33:41.511904 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:41.511907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:41.511976 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:41.512032 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:33:41.512225 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:41.512751 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:41.513608 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:41.513813 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:41.513892 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:41.514155 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:33:41.514220 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:33:41.514274 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:41.514351 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 02:33:41.514417 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":31}
I20260927 02:33:41.514567 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:41.511446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33553 (local address 127.0.98.2:57507)
0927 02:33:41.511715 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:41.511726 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:41.511821 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:41.512048 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:41.512051 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:41.512057 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:33:41.512143 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:41.512148 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:41.512841 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:41.512844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:41.513509 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:41.513518 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:41.513898 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:41.513901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:41.514014 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:41.514049 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:33:41.514284 (+ 235us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:33:41.514289 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:33:41.514344 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 02:33:41.514459 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":42,"thread_start_us":114,"threads_started":1}
I20260927 02:33:41.523962 1947 raft_consensus.cc:1261] T 1494a4c70d954830bb9a1793fe3f5a50 P e62fb61a7e20421caccba2f459dd16fb [term 1 FOLLOWER]: Refusing update from remote peer e0e08aa3ca404cea99e4b5da45d4a9b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:33:41.523970 1667 raft_consensus.cc:1261] T 1494a4c70d954830bb9a1793fe3f5a50 P 3e020b8f631243e2a93860e0fb7bdff7 [term 1 FOLLOWER]: Refusing update from remote peer e0e08aa3ca404cea99e4b5da45d4a9b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:33:41.524431 2023 consensus_queue.cc:1035] T 1494a4c70d954830bb9a1793fe3f5a50 P e0e08aa3ca404cea99e4b5da45d4a9b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e62fb61a7e20421caccba2f459dd16fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:33:41.524694 2062 consensus_queue.cc:1035] T 1494a4c70d954830bb9a1793fe3f5a50 P e0e08aa3ca404cea99e4b5da45d4a9b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e020b8f631243e2a93860e0fb7bdff7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:33:41.563220 1508 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:33:41.563359 1508 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:33:41.565097 1508 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0c09af2b5fc64aa3bf8204902b896b0a: Sending DeleteTablet for 3 replicas of tablet 1494a4c70d954830bb9a1793fe3f5a50
I20260927 02:33:41.565585 1927 tablet_service.cc:1514] Processing DeleteTablet for tablet 1494a4c70d954830bb9a1793fe3f5a50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:33:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45054
I20260927 02:33:41.565618 1647 tablet_service.cc:1514] Processing DeleteTablet for tablet 1494a4c70d954830bb9a1793fe3f5a50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:33:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48260
I20260927 02:33:41.565735 2040 tablet_replica.cc:330] T 1494a4c70d954830bb9a1793fe3f5a50 P e62fb61a7e20421caccba2f459dd16fb: stopping tablet replica
I20260927 02:33:41.565833 2040 raft_consensus.cc:2229] T 1494a4c70d954830bb9a1793fe3f5a50 P e62fb61a7e20421caccba2f459dd16fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:33:41.565936 2040 raft_consensus.cc:2258] T 1494a4c70d954830bb9a1793fe3f5a50 P e62fb61a7e20421caccba2f459dd16fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:33:41.566084 2039 tablet_replica.cc:330] T 1494a4c70d954830bb9a1793fe3f5a50 P 3e020b8f631243e2a93860e0fb7bdff7: stopping tablet replica
I20260927 02:33:41.566190 2039 raft_consensus.cc:2229] T 1494a4c70d954830bb9a1793fe3f5a50 P 3e020b8f631243e2a93860e0fb7bdff7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:33:41.566275 2039 raft_consensus.cc:2258] T 1494a4c70d954830bb9a1793fe3f5a50 P 3e020b8f631243e2a93860e0fb7bdff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:33:41.566478 1786 tablet_service.cc:1514] Processing DeleteTablet for tablet 1494a4c70d954830bb9a1793fe3f5a50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:33:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37272
I20260927 02:33:41.566650 2038 tablet_replica.cc:330] T 1494a4c70d954830bb9a1793fe3f5a50 P e0e08aa3ca404cea99e4b5da45d4a9b0: stopping tablet replica
I20260927 02:33:41.566758 2038 raft_consensus.cc:2229] T 1494a4c70d954830bb9a1793fe3f5a50 P e0e08aa3ca404cea99e4b5da45d4a9b0 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:33:41.566912 2038 raft_consensus.cc:2258] T 1494a4c70d954830bb9a1793fe3f5a50 P e0e08aa3ca404cea99e4b5da45d4a9b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:33:41.567028 2040 ts_tablet_manager.cc:1905] T 1494a4c70d954830bb9a1793fe3f5a50 P e62fb61a7e20421caccba2f459dd16fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:33:41.567551 2038 ts_tablet_manager.cc:1905] T 1494a4c70d954830bb9a1793fe3f5a50 P e0e08aa3ca404cea99e4b5da45d4a9b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:33:41.569362 2040 ts_tablet_manager.cc:1918] T 1494a4c70d954830bb9a1793fe3f5a50 P e62fb61a7e20421caccba2f459dd16fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:33:41.569428 2040 log.cc:1196] T 1494a4c70d954830bb9a1793fe3f5a50 P e62fb61a7e20421caccba2f459dd16fb: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-2/wal/wals/1494a4c70d954830bb9a1793fe3f5a50
I20260927 02:33:41.569643 2040 ts_tablet_manager.cc:1939] T 1494a4c70d954830bb9a1793fe3f5a50 P e62fb61a7e20421caccba2f459dd16fb: Deleting consensus metadata
I20260927 02:33:41.570205 1497 catalog_manager.cc:4841] TS e62fb61a7e20421caccba2f459dd16fb (127.0.98.3:44251): tablet 1494a4c70d954830bb9a1793fe3f5a50 (table test-table [id=249c233f52d64732aaf0698e4bd1dbed]) successfully deleted
I20260927 02:33:41.570255 2038 ts_tablet_manager.cc:1918] T 1494a4c70d954830bb9a1793fe3f5a50 P e0e08aa3ca404cea99e4b5da45d4a9b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:33:41.570323 2038 log.cc:1196] T 1494a4c70d954830bb9a1793fe3f5a50 P e0e08aa3ca404cea99e4b5da45d4a9b0: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-1/wal/wals/1494a4c70d954830bb9a1793fe3f5a50
I20260927 02:33:41.570592 2038 ts_tablet_manager.cc:1939] T 1494a4c70d954830bb9a1793fe3f5a50 P e0e08aa3ca404cea99e4b5da45d4a9b0: Deleting consensus metadata
I20260927 02:33:41.571234 1494 catalog_manager.cc:4841] TS e0e08aa3ca404cea99e4b5da45d4a9b0 (127.0.98.2:33553): tablet 1494a4c70d954830bb9a1793fe3f5a50 (table test-table [id=249c233f52d64732aaf0698e4bd1dbed]) successfully deleted
I20260927 02:33:41.571595 2039 ts_tablet_manager.cc:1905] T 1494a4c70d954830bb9a1793fe3f5a50 P 3e020b8f631243e2a93860e0fb7bdff7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:33:41.573602 2039 ts_tablet_manager.cc:1918] T 1494a4c70d954830bb9a1793fe3f5a50 P 3e020b8f631243e2a93860e0fb7bdff7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:33:41.573688 2039 log.cc:1196] T 1494a4c70d954830bb9a1793fe3f5a50 P 3e020b8f631243e2a93860e0fb7bdff7: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476410933508-392-0/minicluster-data/ts-0/wal/wals/1494a4c70d954830bb9a1793fe3f5a50
I20260927 02:33:41.573906 2039 ts_tablet_manager.cc:1939] T 1494a4c70d954830bb9a1793fe3f5a50 P 3e020b8f631243e2a93860e0fb7bdff7: Deleting consensus metadata
I20260927 02:33:41.574520 1497 catalog_manager.cc:4841] TS 3e020b8f631243e2a93860e0fb7bdff7 (127.0.98.1:35549): tablet 1494a4c70d954830bb9a1793fe3f5a50 (table test-table [id=249c233f52d64732aaf0698e4bd1dbed]) successfully deleted
Sep 27 02:33:41 dist-test-slave-grlk krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476421, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:33:41.582840 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:41.573061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48272 (local address 127.0.98.1:35549)
0927 02:33:41.573189 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:41.573194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:41.573452 (+ 258us) server_negotiation.cc:407] Connection header received
0927 02:33:41.573591 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:41.573594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:41.573653 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:41.573726 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:41.574571 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:41.575364 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:41.576145 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:41.576474 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:41.576576 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:41.579644 (+ 3068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:41.579652 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:41.579663 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:41.579684 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:41.581179 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:41.581644 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:41.581649 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:41.581650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:41.581720 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:41.582053 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:41.582056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:41.582057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:41.582264 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:41.582416 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:41.582634 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 02:33:41.582721 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 02:33:41.583385 1707 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48272
I20260927 02:33:41.591342 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:41.584915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45280 (local address 127.0.98.62:43451)
0927 02:33:41.585030 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:41.585034 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:41.585168 (+ 134us) server_negotiation.cc:407] Connection header received
0927 02:33:41.585267 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:41.585270 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:41.585313 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:41.585399 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:41.585936 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:41.586432 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:41.587147 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:41.587362 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:41.587441 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:41.588597 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:41.588600 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:41.588603 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:41.588627 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:41.589911 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:41.590265 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:41.590269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:41.590270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:41.590337 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:41.590557 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:41.590560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:41.590562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:41.590743 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:41.590853 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:41.591062 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 02:33:41.591204 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 02:33:41.591648 1508 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45280
I20260927 02:33:41.592685 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 1582
I20260927 02:33:41.598570 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 1722
I20260927 02:33:41.604102 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 1862
I20260927 02:33:41.608489 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 1478
2026-09-27T02:33:41Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5195 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:33:41 dist-test-slave-grlk krb5kdc[2073](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:33:41 dist-test-slave-grlk krb5kdc[2073](info): set up 2 sockets
Sep 27 02:33:41 dist-test-slave-grlk krb5kdc[2073](info): commencing operation
krb5kdc: starting...
W20260927 02:33:42.548592 392 mini_kdc.cc:121] Time spent starting KDC: real 0.920s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/krb5kdc/test-user.keytab.
Sep 27 02:33:42 dist-test-slave-grlk krb5kdc[2073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:33:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:33:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:33:42.654053 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38251
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:33:42.754433 2086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:33:42.754662 2086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:33:42.754709 2086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:33:42.758551 2086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:33:42.758596 2086 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 02:33:42.758617 2086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:33:42.758641 2086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:33:42.758662 2086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:33:42.758690 2086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:33:42.763635 2086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38251
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:33:42.764573 2086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:33:42.765542 2086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:33:42.773229 2094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:42.773250 2091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:42.773243 2092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:42.773499 2086 server_base.cc:1034] running on GCE node
I20260927 02:33:42.773983 2086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:33:42.774924 2086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:33:42.776078 2086 hybrid_clock.cc:648] HybridClock initialized: now 1790476422776058 us; error 28 us; skew 500 ppm
Sep 27 02:33:42 dist-test-slave-grlk krb5kdc[2073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476422, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:33:42.778456 2086 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:33:42.779229 2086 webserver.cc:460] Webserver started at http://127.0.98.62:37653/ using document root <none> and password file <none>
I20260927 02:33:42.779702 2086 fs_manager.cc:359] Metadata directory not provided
I20260927 02:33:42.779762 2086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:33:42.779925 2086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:33:42.781837 2086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "307e7dee3cca4210b78b3e0eeba083a4"
format_stamp: "Formatted at 2026-09-27 02:33:42 on dist-test-slave-grlk"
server_key: "886bf649de0f8af0a0bf29ac88193394"
server_key_iv: "4c425a0da36f3aa1ade777e2619df3b1"
server_key_version: "encryptionkey@0"
I20260927 02:33:42.782321 2086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "307e7dee3cca4210b78b3e0eeba083a4"
format_stamp: "Formatted at 2026-09-27 02:33:42 on dist-test-slave-grlk"
server_key: "886bf649de0f8af0a0bf29ac88193394"
server_key_iv: "4c425a0da36f3aa1ade777e2619df3b1"
server_key_version: "encryptionkey@0"
I20260927 02:33:42.785676 2086 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:33:42.787796 2101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:42.788913 2086 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:33:42.789036 2086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "307e7dee3cca4210b78b3e0eeba083a4"
format_stamp: "Formatted at 2026-09-27 02:33:42 on dist-test-slave-grlk"
server_key: "886bf649de0f8af0a0bf29ac88193394"
server_key_iv: "4c425a0da36f3aa1ade777e2619df3b1"
server_key_version: "encryptionkey@0"
I20260927 02:33:42.789127 2086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:33:42.801635 2086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:33:42.803947 2086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:33:42.804082 2086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:33:42.810973 2086 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38251
I20260927 02:33:42.810985 2163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38251 every 8 connection(s)
I20260927 02:33:42.811789 2086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:33:42.814258 2164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:33:42.818462 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 2086
I20260927 02:33:42.818574 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:33:42.818826 392 external_mini_cluster.cc:1428] Setting key a241dc63f425a0da8a950386a23319be
I20260927 02:33:42.819845 2164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 307e7dee3cca4210b78b3e0eeba083a4: Bootstrap starting.
I20260927 02:33:42.822098 2164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 307e7dee3cca4210b78b3e0eeba083a4: Neither blocks nor log segments found. Creating new log.
I20260927 02:33:42.823446 2164 log.cc:824] T 00000000000000000000000000000000 P 307e7dee3cca4210b78b3e0eeba083a4: Log is configured to *not* fsync() on all Append() calls
I20260927 02:33:42.825647 2164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 307e7dee3cca4210b78b3e0eeba083a4: No bootstrap required, opened a new log
Sep 27 02:33:42 dist-test-slave-grlk krb5kdc[2073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:33:42.829700 2164 raft_consensus.cc:348] T 00000000000000000000000000000000 P 307e7dee3cca4210b78b3e0eeba083a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307e7dee3cca4210b78b3e0eeba083a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38251 } }
I20260927 02:33:42.829926 2164 raft_consensus.cc:374] T 00000000000000000000000000000000 P 307e7dee3cca4210b78b3e0eeba083a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:33:42.829983 2164 raft_consensus.cc:729] T 00000000000000000000000000000000 P 307e7dee3cca4210b78b3e0eeba083a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 307e7dee3cca4210b78b3e0eeba083a4, State: Initialized, Role: FOLLOWER
I20260927 02:33:42.830399 2164 consensus_queue.cc:260] T 00000000000000000000000000000000 P 307e7dee3cca4210b78b3e0eeba083a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307e7dee3cca4210b78b3e0eeba083a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38251 } }
I20260927 02:33:42.830569 2164 raft_consensus.cc:388] T 00000000000000000000000000000000 P 307e7dee3cca4210b78b3e0eeba083a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:33:42.830646 2164 raft_consensus.cc:482] T 00000000000000000000000000000000 P 307e7dee3cca4210b78b3e0eeba083a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:33:42.830744 2164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 307e7dee3cca4210b78b3e0eeba083a4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:33:42.831786 2164 raft_consensus.cc:504] T 00000000000000000000000000000000 P 307e7dee3cca4210b78b3e0eeba083a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307e7dee3cca4210b78b3e0eeba083a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38251 } }
I20260927 02:33:42.832127 2164 leader_election.cc:302] T 00000000000000000000000000000000 P 307e7dee3cca4210b78b3e0eeba083a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 307e7dee3cca4210b78b3e0eeba083a4; no voters:
I20260927 02:33:42.832419 2164 leader_election.cc:288] T 00000000000000000000000000000000 P 307e7dee3cca4210b78b3e0eeba083a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:33:42.832643 2169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 307e7dee3cca4210b78b3e0eeba083a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:33:42.832969 2169 raft_consensus.cc:686] T 00000000000000000000000000000000 P 307e7dee3cca4210b78b3e0eeba083a4 [term 1 LEADER]: Becoming Leader. State: Replica: 307e7dee3cca4210b78b3e0eeba083a4, State: Running, Role: LEADER
I20260927 02:33:42.833407 2169 consensus_queue.cc:237] T 00000000000000000000000000000000 P 307e7dee3cca4210b78b3e0eeba083a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307e7dee3cca4210b78b3e0eeba083a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38251 } }
I20260927 02:33:42.833657 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:42.820097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33314 (local address 127.0.98.62:38251)
0927 02:33:42.820457 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:42.820471 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:42.820502 (+ 31us) server_negotiation.cc:407] Connection header received
0927 02:33:42.821151 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:42.821169 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:42.821483 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:42.821745 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:42.823054 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:42.824224 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:42.825357 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:42.825847 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:42.825959 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:42.828617 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:42.828622 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:42.828634 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:42.828659 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:42.830247 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:42.830713 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:42.830718 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:42.830724 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:42.830887 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:42.831184 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:42.831187 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:42.831189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:42.831679 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:42.831880 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:42.832184 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 02:33:42.832604 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":88,"threads_started":1}
I20260927 02:33:42.833976 2164 sys_catalog.cc:564] T 00000000000000000000000000000000 P 307e7dee3cca4210b78b3e0eeba083a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:33:42.834563 2169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 307e7dee3cca4210b78b3e0eeba083a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "307e7dee3cca4210b78b3e0eeba083a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307e7dee3cca4210b78b3e0eeba083a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38251 } } }
I20260927 02:33:42.834677 2169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 307e7dee3cca4210b78b3e0eeba083a4 [sys.catalog]: This master's current role is: LEADER
I20260927 02:33:42.834851 2169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 307e7dee3cca4210b78b3e0eeba083a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 307e7dee3cca4210b78b3e0eeba083a4. Latest consensus state: current_term: 1 leader_uuid: "307e7dee3cca4210b78b3e0eeba083a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307e7dee3cca4210b78b3e0eeba083a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38251 } } }
I20260927 02:33:42.834930 2169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 307e7dee3cca4210b78b3e0eeba083a4 [sys.catalog]: This master's current role is: LEADER
W20260927 02:33:42.838518 2184 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 307e7dee3cca4210b78b3e0eeba083a4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:33:42.838583 2184 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:33:42.838661 2175 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:33:42.839530 2175 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:33:42.844724 2175 catalog_manager.cc:1347] Generated new cluster ID: 093adf29c79a420faafa7a531353c46e
I20260927 02:33:42.844789 2175 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:33:42.860572 2175 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:33:42.861819 2175 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:33:42.869547 2175 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 307e7dee3cca4210b78b3e0eeba083a4: Generated new TSK 0
I20260927 02:33:42.870252 2175 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:33:42.897596 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38251
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:33:43.005198 2190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:33:43.005455 2190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:33:43.005506 2190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:33:43.009617 2190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:33:43.009681 2190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:33:43.009720 2190 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 02:33:43.009747 2190 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:33:43.009797 2190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:33:43.015072 2190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:33:43.016103 2190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:33:43.017112 2190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:33:43.026019 2198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:43.026129 2190 server_base.cc:1034] running on GCE node
W20260927 02:33:43.026072 2196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:43.026058 2195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:43.026633 2190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:33:43.027185 2190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:33:43.028323 2190 hybrid_clock.cc:648] HybridClock initialized: now 1790476423028298 us; error 36 us; skew 500 ppm
Sep 27 02:33:43 dist-test-slave-grlk krb5kdc[2073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:33:43.030776 2190 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:33:43.031558 2190 webserver.cc:460] Webserver started at http://127.0.98.1:42757/ using document root <none> and password file <none>
I20260927 02:33:43.032052 2190 fs_manager.cc:359] Metadata directory not provided
I20260927 02:33:43.032109 2190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:33:43.032312 2190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:33:43.034289 2190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-0/data/instance:
uuid: "f0d8dd12754e439ca34a41122f43de11"
format_stamp: "Formatted at 2026-09-27 02:33:43 on dist-test-slave-grlk"
server_key: "3fef07b27faa85495a5ed8709e7c5966"
server_key_iv: "50089077c54b918da340fb377e7378ee"
server_key_version: "encryptionkey@0"
I20260927 02:33:43.034834 2190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f0d8dd12754e439ca34a41122f43de11"
format_stamp: "Formatted at 2026-09-27 02:33:43 on dist-test-slave-grlk"
server_key: "3fef07b27faa85495a5ed8709e7c5966"
server_key_iv: "50089077c54b918da340fb377e7378ee"
server_key_version: "encryptionkey@0"
I20260927 02:33:43.038275 2190 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:33:43.040439 2205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:43.041584 2190 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:33:43.041690 2190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-0/wal
uuid: "f0d8dd12754e439ca34a41122f43de11"
format_stamp: "Formatted at 2026-09-27 02:33:43 on dist-test-slave-grlk"
server_key: "3fef07b27faa85495a5ed8709e7c5966"
server_key_iv: "50089077c54b918da340fb377e7378ee"
server_key_version: "encryptionkey@0"
I20260927 02:33:43.041775 2190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:33:43.054020 2190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:33:43.056746 2190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:33:43.056895 2190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:33:43.058205 2190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:33:43.058251 2190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:43.058300 2190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:33:43.058327 2190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 02:33:43 dist-test-slave-grlk krb5kdc[2073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:33:43.067874 2190 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40513
I20260927 02:33:43.068022 2320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40513 every 8 connection(s)
I20260927 02:33:43.068771 2190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:33:43.070703 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:43.058821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43025 (local address 127.0.98.62:38251)
0927 02:33:43.058961 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:43.058966 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:43.059723 (+ 757us) server_negotiation.cc:407] Connection header received
0927 02:33:43.060659 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:43.060663 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:43.060718 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:43.060773 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:43.062176 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.062759 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.063921 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.064165 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.064253 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:43.067317 (+ 3064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:43.067321 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:43.067323 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:43.067350 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:43.068771 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:43.069401 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:43.069405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:43.069407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:43.069499 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:43.069820 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:43.069823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:43.069824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:43.070000 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:43.070076 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:43.070468 (+ 392us) server_negotiation.cc:299] Negotiation successful
0927 02:33:43.070575 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 02:33:43.071437 2214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:43.059167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38251 (local address 127.0.98.1:43025)
0927 02:33:43.059571 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:43.059618 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:43.060429 (+ 811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:43.061166 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:43.061174 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:43.061535 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:33:43.062038 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:43.062052 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.062877 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.062880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:43.063811 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:43.063820 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.064309 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.064313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:43.064476 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:43.065041 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:33:43.065062 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:33:43.067145 (+ 2083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:33:43.068882 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:43.068887 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:43.068900 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:43.069272 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:43.069606 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:43.069609 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:43.069610 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:43.069732 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:43.070076 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:33:43.070082 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:33:43.070288 (+ 206us) client_negotiation.cc:769] Sending connection context
0927 02:33:43.070424 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 02:33:43.070668 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":92,"threads_started":1}
I20260927 02:33:43.072479 2321 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38251
I20260927 02:33:43.072682 2321 heartbeater.cc:464] Registering TS with master...
I20260927 02:33:43.073163 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 2190
I20260927 02:33:43.073197 2321 heartbeater.cc:511] Master 127.0.98.62:38251 requested a full tablet report, sending...
I20260927 02:33:43.073266 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:33:43.073537 392 external_mini_cluster.cc:1428] Setting key 15c52d985580af637074f25ab456734c
I20260927 02:33:43.074720 2117 ts_manager.cc:194] Registered new tserver with Master: f0d8dd12754e439ca34a41122f43de11 (127.0.98.1:40513)
I20260927 02:33:43.076157 2117 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43025
I20260927 02:33:43.081945 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:43.075218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43183 (local address 127.0.98.62:38251)
0927 02:33:43.075329 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:43.075333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:43.075345 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:33:43.075375 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:43.075378 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:43.075417 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:43.075483 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:43.075994 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.076692 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.077854 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.078097 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.078188 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:43.079234 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:43.079238 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:43.079240 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:43.079262 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:43.080475 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:43.080846 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:43.080849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:43.080851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:43.080921 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:43.081209 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:43.081213 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:43.081214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:43.081376 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:43.081456 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:43.081688 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 02:33:43.081787 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 02:33:43.081957 2327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:43.074771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38251 (local address 127.0.98.1:43183)
0927 02:33:43.075011 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:43.075025 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:43.075113 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:43.075576 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:43.075579 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:43.075775 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:33:43.075881 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:43.075887 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.076869 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.076872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:43.077733 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:43.077742 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.078201 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.078205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:43.078292 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:43.078753 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:33:43.078763 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:33:43.079109 (+ 346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:33:43.080574 (+ 1465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:43.080577 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:43.080579 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:43.080762 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:43.081049 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:43.081052 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:43.081054 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:43.081135 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:43.081466 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:33:43.081469 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:33:43.081550 (+ 81us) client_negotiation.cc:769] Sending connection context
0927 02:33:43.081633 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 02:33:43.081767 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":23,"thread_start_us":79,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:33:43.097585 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38251
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:33:43.203481 2330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:33:43.203755 2330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:33:43.203802 2330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:33:43.207778 2330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:33:43.207834 2330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:33:43.207872 2330 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 02:33:43.207898 2330 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:33:43.207949 2330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:33:43.213035 2330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:33:43.214013 2330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:33:43.214947 2330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:33:43.223953 2338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:43.224105 2336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:43.223953 2335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:43.224310 2330 server_base.cc:1034] running on GCE node
I20260927 02:33:43.224655 2330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:33:43.225245 2330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:33:43.226388 2330 hybrid_clock.cc:648] HybridClock initialized: now 1790476423226374 us; error 27 us; skew 500 ppm
Sep 27 02:33:43 dist-test-slave-grlk krb5kdc[2073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:33:43.232318 2330 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:33:43.233139 2330 webserver.cc:460] Webserver started at http://127.0.98.2:43341/ using document root <none> and password file <none>
I20260927 02:33:43.233656 2330 fs_manager.cc:359] Metadata directory not provided
I20260927 02:33:43.233718 2330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:33:43.233908 2330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:33:43.235884 2330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-1/data/instance:
uuid: "e31b21f7fd594078ae3066bbb49a09c1"
format_stamp: "Formatted at 2026-09-27 02:33:43 on dist-test-slave-grlk"
server_key: "2723c47cd3d47391af4dd299b08c1661"
server_key_iv: "8dfd4fc212064fade2246cdd0449bd29"
server_key_version: "encryptionkey@0"
I20260927 02:33:43.236374 2330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e31b21f7fd594078ae3066bbb49a09c1"
format_stamp: "Formatted at 2026-09-27 02:33:43 on dist-test-slave-grlk"
server_key: "2723c47cd3d47391af4dd299b08c1661"
server_key_iv: "8dfd4fc212064fade2246cdd0449bd29"
server_key_version: "encryptionkey@0"
I20260927 02:33:43.239637 2330 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260927 02:33:43.241781 2345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:43.242848 2330 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:33:43.242946 2330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-1/wal
uuid: "e31b21f7fd594078ae3066bbb49a09c1"
format_stamp: "Formatted at 2026-09-27 02:33:43 on dist-test-slave-grlk"
server_key: "2723c47cd3d47391af4dd299b08c1661"
server_key_iv: "8dfd4fc212064fade2246cdd0449bd29"
server_key_version: "encryptionkey@0"
I20260927 02:33:43.243028 2330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:33:43.255265 2330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:33:43.258082 2330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:33:43.258242 2330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:33:43.259563 2330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:33:43.259617 2330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:43.259666 2330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:33:43.259696 2330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 02:33:43 dist-test-slave-grlk krb5kdc[2073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:33:43.269217 2330 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45769
I20260927 02:33:43.269249 2460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45769 every 8 connection(s)
I20260927 02:33:43.270421 2330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:33:43.272962 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 2330
I20260927 02:33:43.273072 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:33:43.273329 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:43.260270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43685 (local address 127.0.98.62:38251)
0927 02:33:43.260406 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:43.260410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:43.261239 (+ 829us) server_negotiation.cc:407] Connection header received
0927 02:33:43.262184 (+ 945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:43.262188 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:43.262241 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:43.262310 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:43.263764 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.264562 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.265843 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.266072 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.266159 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:43.268607 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:43.268612 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:43.268615 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:43.268647 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:43.270455 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:43.271498 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:43.271501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:43.271503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:43.271615 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:43.272123 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:43.272128 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:43.272130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:43.272355 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:43.272456 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:43.273012 (+ 556us) server_negotiation.cc:299] Negotiation successful
0927 02:33:43.273142 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 02:33:43.273352 392 external_mini_cluster.cc:1428] Setting key 0d09ee56f9fe59bb8567f8b39aa63c4b
I20260927 02:33:43.274255 2354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:43.260644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38251 (local address 127.0.98.2:43685)
0927 02:33:43.261082 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:43.261129 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:43.261984 (+ 855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:43.262619 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:43.262628 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:43.263035 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:33:43.263636 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:43.263655 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.264676 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.264680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:43.265736 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:43.265746 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.266203 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.266207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:43.266371 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:43.266965 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:33:43.266981 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:33:43.268423 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:33:43.270742 (+ 2319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:43.270750 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:43.270771 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:43.271329 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:43.271761 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:43.271765 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:43.271767 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:43.271975 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:43.272489 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:33:43.272494 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:33:43.272727 (+ 233us) client_negotiation.cc:769] Sending connection context
0927 02:33:43.272918 (+ 191us) client_negotiation.cc:240] Negotiation successful
0927 02:33:43.273305 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":91,"threads_started":1}
I20260927 02:33:43.275393 2461 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38251
I20260927 02:33:43.275534 2461 heartbeater.cc:464] Registering TS with master...
I20260927 02:33:43.276026 2461 heartbeater.cc:511] Master 127.0.98.62:38251 requested a full tablet report, sending...
I20260927 02:33:43.277063 2118 ts_manager.cc:194] Registered new tserver with Master: e31b21f7fd594078ae3066bbb49a09c1 (127.0.98.2:45769)
I20260927 02:33:43.277621 2118 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43685
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:33:43.284471 2468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:43.277990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38251 (local address 127.0.98.2:34257)
0927 02:33:43.278254 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:43.278265 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:43.278347 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:43.278635 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:43.278637 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:43.278802 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:33:43.278886 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:43.278891 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.279855 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.279858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:43.280393 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:43.280400 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.280923 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.280926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:43.281033 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:43.281494 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:33:43.281504 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:33:43.281776 (+ 272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:33:43.283275 (+ 1499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:43.283278 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:43.283279 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:43.283492 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:43.283693 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:43.283695 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:43.283696 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:43.283760 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:43.284072 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:33:43.284074 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:33:43.284161 (+ 87us) client_negotiation.cc:769] Sending connection context
0927 02:33:43.284253 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 02:33:43.284350 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":73,"threads_started":1}
I20260927 02:33:43.284476 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:43.278069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34257 (local address 127.0.98.62:38251)
0927 02:33:43.278169 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:43.278173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:43.278327 (+ 154us) server_negotiation.cc:407] Connection header received
0927 02:33:43.278448 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:43.278451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:43.278488 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:43.278536 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:43.278987 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.279739 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.280499 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.280828 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.280952 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:43.281902 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:43.281905 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:43.281908 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:43.281933 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:43.283183 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:43.283564 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:43.283567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:43.283568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:43.283633 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:43.283815 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:43.283816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:43.283818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:43.283984 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:43.284081 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:43.284299 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 02:33:43.284379 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:33:43.297838 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38251
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:33:43.402024 2470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:33:43.402266 2470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:33:43.402311 2470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:33:43.406235 2470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:33:43.406284 2470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:33:43.406327 2470 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 02:33:43.406354 2470 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:33:43.406401 2470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:33:43.411463 2470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:33:43.412384 2470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:33:43.413321 2470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:33:43.422364 2470 server_base.cc:1034] running on GCE node
W20260927 02:33:43.422328 2478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:43.422317 2475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:43.422307 2476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:43.422854 2470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:33:43.423466 2470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:33:43.424642 2470 hybrid_clock.cc:648] HybridClock initialized: now 1790476423424628 us; error 25 us; skew 500 ppm
Sep 27 02:33:43 dist-test-slave-grlk krb5kdc[2073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:33:43.427090 2470 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:33:43.427865 2470 webserver.cc:460] Webserver started at http://127.0.98.3:35657/ using document root <none> and password file <none>
I20260927 02:33:43.428341 2470 fs_manager.cc:359] Metadata directory not provided
I20260927 02:33:43.428400 2470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:33:43.428591 2470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:33:43.430584 2470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-2/data/instance:
uuid: "74fd0b61786e4ef2abc5c35685bc05f7"
format_stamp: "Formatted at 2026-09-27 02:33:43 on dist-test-slave-grlk"
server_key: "5ed988749846c31a50abaab3acce7a0b"
server_key_iv: "8bed79133315ad6c93692bfc3d15ceb8"
server_key_version: "encryptionkey@0"
I20260927 02:33:43.431130 2470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-2/wal/instance:
uuid: "74fd0b61786e4ef2abc5c35685bc05f7"
format_stamp: "Formatted at 2026-09-27 02:33:43 on dist-test-slave-grlk"
server_key: "5ed988749846c31a50abaab3acce7a0b"
server_key_iv: "8bed79133315ad6c93692bfc3d15ceb8"
server_key_version: "encryptionkey@0"
I20260927 02:33:43.434464 2470 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:33:43.436594 2485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:43.437668 2470 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:33:43.437763 2470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-2/wal
uuid: "74fd0b61786e4ef2abc5c35685bc05f7"
format_stamp: "Formatted at 2026-09-27 02:33:43 on dist-test-slave-grlk"
server_key: "5ed988749846c31a50abaab3acce7a0b"
server_key_iv: "8bed79133315ad6c93692bfc3d15ceb8"
server_key_version: "encryptionkey@0"
I20260927 02:33:43.437845 2470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:33:43.454846 2470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:33:43.457661 2470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:33:43.457820 2470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:33:43.459146 2470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:33:43.459194 2470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:43.459244 2470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:33:43.459272 2470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 02:33:43 dist-test-slave-grlk krb5kdc[2073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:33:43.468777 2470 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34811
I20260927 02:33:43.468838 2600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34811 every 8 connection(s)
I20260927 02:33:43.469691 2470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:33:43.472190 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:43.459880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35551 (local address 127.0.98.62:38251)
0927 02:33:43.460003 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:43.460006 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:43.460989 (+ 983us) server_negotiation.cc:407] Connection header received
0927 02:33:43.462022 (+ 1033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:43.462026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:43.462089 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:43.462182 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:43.463622 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.464411 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.465721 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.466068 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.466187 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:43.468857 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:43.468862 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:43.468865 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:43.468957 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:43.470462 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:43.470954 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:43.470957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:43.470959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:43.471043 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:43.471296 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:43.471298 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:43.471299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:43.471477 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:43.471554 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:43.471979 (+ 425us) server_negotiation.cc:299] Negotiation successful
0927 02:33:43.472077 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 02:33:43.472707 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 2470
I20260927 02:33:43.472806 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:33:43.473050 2494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:43.460362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38251 (local address 127.0.98.3:35551)
0927 02:33:43.460799 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:43.460840 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:43.461749 (+ 909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:43.462365 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:43.462375 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:43.462819 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:33:43.463436 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:43.463454 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.464543 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.464547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:43.465596 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:43.465606 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.466218 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.466222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:43.466409 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:43.466943 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:33:43.466965 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:33:43.468696 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:33:43.470574 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:43.470579 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:43.470591 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:43.470868 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:43.471117 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:43.471119 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:43.471120 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:43.471229 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:43.471550 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:33:43.471554 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:33:43.471762 (+ 208us) client_negotiation.cc:769] Sending connection context
0927 02:33:43.471903 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 02:33:43.472121 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":115,"threads_started":1}
I20260927 02:33:43.473122 392 external_mini_cluster.cc:1428] Setting key 74f3a25eb26ce9307a81809986e45021
I20260927 02:33:43.474335 2601 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38251
I20260927 02:33:43.474485 2601 heartbeater.cc:464] Registering TS with master...
I20260927 02:33:43.474931 2601 heartbeater.cc:511] Master 127.0.98.62:38251 requested a full tablet report, sending...
I20260927 02:33:43.475924 2118 ts_manager.cc:194] Registered new tserver with Master: 74fd0b61786e4ef2abc5c35685bc05f7 (127.0.98.3:34811)
I20260927 02:33:43.476572 2118 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35551
I20260927 02:33:43.482167 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:43.476548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49831 (local address 127.0.98.62:38251)
0927 02:33:43.476659 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:43.476663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:43.476677 (+ 14us) server_negotiation.cc:407] Connection header received
0927 02:33:43.476709 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:43.476712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:43.476748 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:43.476828 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:43.477263 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.477768 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.478527 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.478741 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.478818 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:43.479735 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:43.479739 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:43.479742 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:43.479764 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:43.480765 (+ 1001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:43.481173 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:43.481176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:43.481178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:43.481251 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:43.481478 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:43.481481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:43.481483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:43.481639 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:43.481717 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:43.481944 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 02:33:43.482037 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 02:33:43.482179 2607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:43.476288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38251 (local address 127.0.98.3:49831)
0927 02:33:43.476543 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:43.476557 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:43.476639 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:43.476846 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:43.476849 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:43.477067 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:33:43.477163 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:43.477169 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.477878 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.477882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:43.478430 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:43.478437 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.478829 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.478833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:43.478916 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:43.479325 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:33:43.479335 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:33:43.479621 (+ 286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:33:43.480857 (+ 1236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:43.480860 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:43.480862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:43.481084 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:43.481322 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:43.481325 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:43.481327 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:43.481399 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:43.481718 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:33:43.481721 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:33:43.481800 (+ 79us) client_negotiation.cc:769] Sending connection context
0927 02:33:43.481878 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 02:33:43.482002 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":81,"threads_started":1}
I20260927 02:33:43.486487 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 02:33:43 dist-test-slave-grlk krb5kdc[2073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 02:33:43 dist-test-slave-grlk krb5kdc[2073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:33:43.508332 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:43.501633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33316 (local address 127.0.98.62:38251)
0927 02:33:43.501748 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:43.501751 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:43.501761 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:33:43.501793 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:43.501795 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:43.501829 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:43.501887 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:43.502548 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.503050 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.503784 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.503976 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.504040 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:43.506009 (+ 1969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:43.506012 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:43.506013 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:43.506035 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:43.507238 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:43.507527 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:43.507529 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:43.507531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:43.507593 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:43.507765 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:43.507766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:43.507767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:43.507906 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:43.507988 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:43.508138 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 02:33:43.508220 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 02:33:43.510831 2118 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33316:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:33:43.513716 2118 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:33:43.526981 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:43.521847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43464 (local address 127.0.98.1:40513)
0927 02:33:43.522260 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:43.522493 (+ 233us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:43.522516 (+ 23us) server_negotiation.cc:407] Connection header received
0927 02:33:43.522595 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:43.522600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:43.522766 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:43.522915 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:33:43.523136 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.523885 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.525802 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.526345 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.526474 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:43.526650 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:43.526750 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 02:33:43.526842 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":104,"threads_started":1}
I20260927 02:33:43.527009 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:43.521093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34811 (local address 127.0.0.1:44980)
0927 02:33:43.521405 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:43.521451 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:43.521711 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:43.522684 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:43.522690 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:43.522712 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:33:43.522828 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:43.522834 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.524323 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.524326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:43.525647 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:43.525653 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.526627 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.526631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:43.526717 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:43.526740 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 02:33:43.526787 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 02:33:43.526866 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":75,"thread_start_us":67,"threads_started":1}
I20260927 02:33:43.527138 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:43.522251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53548 (local address 127.0.98.2:45769)
0927 02:33:43.522644 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:43.522655 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:43.522674 (+ 19us) server_negotiation.cc:407] Connection header received
0927 02:33:43.522735 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:43.522740 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:43.522915 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:43.523047 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:33:43.523217 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.524293 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.525583 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.526363 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.526464 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:43.526634 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:43.526936 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 02:33:43.527017 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":100,"threads_started":1}
I20260927 02:33:43.527536 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:43.521222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44980 (local address 127.0.98.3:34811)
0927 02:33:43.522227 (+ 1005us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:43.522238 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:43.522260 (+ 22us) server_negotiation.cc:407] Connection header received
0927 02:33:43.522331 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:43.522336 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:43.522535 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:43.522684 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:33:43.522955 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.524183 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.525765 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.526506 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.527183 (+ 677us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:43.527331 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:43.527375 (+ 44us) server_negotiation.cc:299] Negotiation successful
0927 02:33:43.527439 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":90,"threads_started":1}
I20260927 02:33:43.527761 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:43.521883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45769 (local address 127.0.0.1:53548)
0927 02:33:43.522376 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:43.522388 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:43.522465 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:43.523051 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:43.523054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:43.523069 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:33:43.523136 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:43.523141 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.524394 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.524397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:43.525474 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:43.525483 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.526681 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.526685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:43.526753 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:43.526770 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 02:33:43.526824 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 02:33:43.526874 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"mutex_wait_us":3,"thread_start_us":81,"threads_started":1}
I20260927 02:33:43.527776 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:43.521281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40513 (local address 127.0.0.1:43464)
0927 02:33:43.521931 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:43.521944 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:43.522016 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:43.522890 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:43.522892 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:43.522910 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:33:43.523008 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:43.523015 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.523987 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.523990 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:43.525643 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:43.525653 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.526466 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.526470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:43.526574 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:43.526609 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 02:33:43.526722 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 02:33:43.526840 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"mutex_wait_us":128,"thread_start_us":93,"threads_started":1}
I20260927 02:33:43.530138 2255 tablet_service.cc:1467] Processing CreateTablet for tablet 28d4bbac9dd549309f5cbd3dda6ec647 (DEFAULT_TABLE table=test-table [id=a8278cd8a3f44f8d86f9bd0b4c683606]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:33:43.530257 2535 tablet_service.cc:1467] Processing CreateTablet for tablet 28d4bbac9dd549309f5cbd3dda6ec647 (DEFAULT_TABLE table=test-table [id=a8278cd8a3f44f8d86f9bd0b4c683606]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:33:43.530256 2395 tablet_service.cc:1467] Processing CreateTablet for tablet 28d4bbac9dd549309f5cbd3dda6ec647 (DEFAULT_TABLE table=test-table [id=a8278cd8a3f44f8d86f9bd0b4c683606]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:33:43.531035 2255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28d4bbac9dd549309f5cbd3dda6ec647. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:33:43.531199 2395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28d4bbac9dd549309f5cbd3dda6ec647. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:33:43.531209 2535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28d4bbac9dd549309f5cbd3dda6ec647. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:33:43.537611 2624 tablet_bootstrap.cc:492] T 28d4bbac9dd549309f5cbd3dda6ec647 P f0d8dd12754e439ca34a41122f43de11: Bootstrap starting.
I20260927 02:33:43.538988 2626 tablet_bootstrap.cc:492] T 28d4bbac9dd549309f5cbd3dda6ec647 P 74fd0b61786e4ef2abc5c35685bc05f7: Bootstrap starting.
I20260927 02:33:43.538988 2625 tablet_bootstrap.cc:492] T 28d4bbac9dd549309f5cbd3dda6ec647 P e31b21f7fd594078ae3066bbb49a09c1: Bootstrap starting.
I20260927 02:33:43.539669 2624 tablet_bootstrap.cc:654] T 28d4bbac9dd549309f5cbd3dda6ec647 P f0d8dd12754e439ca34a41122f43de11: Neither blocks nor log segments found. Creating new log.
I20260927 02:33:43.540388 2624 log.cc:824] T 28d4bbac9dd549309f5cbd3dda6ec647 P f0d8dd12754e439ca34a41122f43de11: Log is configured to *not* fsync() on all Append() calls
I20260927 02:33:43.541028 2626 tablet_bootstrap.cc:654] T 28d4bbac9dd549309f5cbd3dda6ec647 P 74fd0b61786e4ef2abc5c35685bc05f7: Neither blocks nor log segments found. Creating new log.
I20260927 02:33:43.541042 2625 tablet_bootstrap.cc:654] T 28d4bbac9dd549309f5cbd3dda6ec647 P e31b21f7fd594078ae3066bbb49a09c1: Neither blocks nor log segments found. Creating new log.
I20260927 02:33:43.541800 2625 log.cc:824] T 28d4bbac9dd549309f5cbd3dda6ec647 P e31b21f7fd594078ae3066bbb49a09c1: Log is configured to *not* fsync() on all Append() calls
I20260927 02:33:43.541800 2626 log.cc:824] T 28d4bbac9dd549309f5cbd3dda6ec647 P 74fd0b61786e4ef2abc5c35685bc05f7: Log is configured to *not* fsync() on all Append() calls
I20260927 02:33:43.542323 2624 tablet_bootstrap.cc:492] T 28d4bbac9dd549309f5cbd3dda6ec647 P f0d8dd12754e439ca34a41122f43de11: No bootstrap required, opened a new log
I20260927 02:33:43.542513 2624 ts_tablet_manager.cc:1397] T 28d4bbac9dd549309f5cbd3dda6ec647 P f0d8dd12754e439ca34a41122f43de11: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 02:33:43.543715 2625 tablet_bootstrap.cc:492] T 28d4bbac9dd549309f5cbd3dda6ec647 P e31b21f7fd594078ae3066bbb49a09c1: No bootstrap required, opened a new log
I20260927 02:33:43.543886 2625 ts_tablet_manager.cc:1397] T 28d4bbac9dd549309f5cbd3dda6ec647 P e31b21f7fd594078ae3066bbb49a09c1: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260927 02:33:43.544057 2626 tablet_bootstrap.cc:492] T 28d4bbac9dd549309f5cbd3dda6ec647 P 74fd0b61786e4ef2abc5c35685bc05f7: No bootstrap required, opened a new log
I20260927 02:33:43.544214 2626 ts_tablet_manager.cc:1397] T 28d4bbac9dd549309f5cbd3dda6ec647 P 74fd0b61786e4ef2abc5c35685bc05f7: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260927 02:33:43.545552 2624 raft_consensus.cc:348] T 28d4bbac9dd549309f5cbd3dda6ec647 P f0d8dd12754e439ca34a41122f43de11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } } peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } }
I20260927 02:33:43.545791 2624 raft_consensus.cc:374] T 28d4bbac9dd549309f5cbd3dda6ec647 P f0d8dd12754e439ca34a41122f43de11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:33:43.545843 2624 raft_consensus.cc:729] T 28d4bbac9dd549309f5cbd3dda6ec647 P f0d8dd12754e439ca34a41122f43de11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0d8dd12754e439ca34a41122f43de11, State: Initialized, Role: FOLLOWER
I20260927 02:33:43.546243 2624 consensus_queue.cc:260] T 28d4bbac9dd549309f5cbd3dda6ec647 P f0d8dd12754e439ca34a41122f43de11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } } peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } }
I20260927 02:33:43.546875 2321 heartbeater.cc:503] Master 127.0.98.62:38251 was elected leader, sending a full tablet report...
I20260927 02:33:43.547227 2624 ts_tablet_manager.cc:1428] T 28d4bbac9dd549309f5cbd3dda6ec647 P f0d8dd12754e439ca34a41122f43de11: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20260927 02:33:43.547544 2625 raft_consensus.cc:348] T 28d4bbac9dd549309f5cbd3dda6ec647 P e31b21f7fd594078ae3066bbb49a09c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } } peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } }
I20260927 02:33:43.547767 2625 raft_consensus.cc:374] T 28d4bbac9dd549309f5cbd3dda6ec647 P e31b21f7fd594078ae3066bbb49a09c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:33:43.547812 2625 raft_consensus.cc:729] T 28d4bbac9dd549309f5cbd3dda6ec647 P e31b21f7fd594078ae3066bbb49a09c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e31b21f7fd594078ae3066bbb49a09c1, State: Initialized, Role: FOLLOWER
I20260927 02:33:43.547943 2626 raft_consensus.cc:348] T 28d4bbac9dd549309f5cbd3dda6ec647 P 74fd0b61786e4ef2abc5c35685bc05f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } } peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } }
I20260927 02:33:43.548167 2626 raft_consensus.cc:374] T 28d4bbac9dd549309f5cbd3dda6ec647 P 74fd0b61786e4ef2abc5c35685bc05f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:33:43.548221 2626 raft_consensus.cc:729] T 28d4bbac9dd549309f5cbd3dda6ec647 P 74fd0b61786e4ef2abc5c35685bc05f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74fd0b61786e4ef2abc5c35685bc05f7, State: Initialized, Role: FOLLOWER
I20260927 02:33:43.548249 2625 consensus_queue.cc:260] T 28d4bbac9dd549309f5cbd3dda6ec647 P e31b21f7fd594078ae3066bbb49a09c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } } peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } }
I20260927 02:33:43.548632 2626 consensus_queue.cc:260] T 28d4bbac9dd549309f5cbd3dda6ec647 P 74fd0b61786e4ef2abc5c35685bc05f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } } peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } }
I20260927 02:33:43.548923 2461 heartbeater.cc:503] Master 127.0.98.62:38251 was elected leader, sending a full tablet report...
I20260927 02:33:43.549453 2625 ts_tablet_manager.cc:1428] T 28d4bbac9dd549309f5cbd3dda6ec647 P e31b21f7fd594078ae3066bbb49a09c1: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260927 02:33:43.550313 2626 ts_tablet_manager.cc:1428] T 28d4bbac9dd549309f5cbd3dda6ec647 P 74fd0b61786e4ef2abc5c35685bc05f7: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260927 02:33:43.550601 2601 heartbeater.cc:503] Master 127.0.98.62:38251 was elected leader, sending a full tablet report...
I20260927 02:33:43.564718 2632 raft_consensus.cc:482] T 28d4bbac9dd549309f5cbd3dda6ec647 P 74fd0b61786e4ef2abc5c35685bc05f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:33:43.564853 2632 raft_consensus.cc:504] T 28d4bbac9dd549309f5cbd3dda6ec647 P 74fd0b61786e4ef2abc5c35685bc05f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } } peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } }
I20260927 02:33:43.565774 2632 leader_election.cc:288] T 28d4bbac9dd549309f5cbd3dda6ec647 P 74fd0b61786e4ef2abc5c35685bc05f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0d8dd12754e439ca34a41122f43de11 (127.0.98.1:40513), e31b21f7fd594078ae3066bbb49a09c1 (127.0.98.2:45769)
I20260927 02:33:43.569590 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:43.566010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41121 (local address 127.0.98.1:40513)
0927 02:33:43.566104 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:43.566109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:43.566209 (+ 100us) server_negotiation.cc:407] Connection header received
0927 02:33:43.566308 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:43.566311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:43.566358 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:43.566433 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:33:43.566681 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.567450 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.568535 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.569142 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.569215 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:43.569262 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:43.569443 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 02:33:43.569492 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 02:33:43.569628 2494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:43.565941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40513 (local address 127.0.98.3:41121)
0927 02:33:43.566120 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:43.566133 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:43.566214 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:43.566445 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:43.566447 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:43.566460 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:33:43.566576 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:43.566582 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.567545 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.567548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:43.568422 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:43.568431 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.569237 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.569241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:43.569338 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:43.569362 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 02:33:43.569434 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 02:33:43.569504 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":4}
W20260927 02:33:43.569877 2322 tablet.cc:2369] T 28d4bbac9dd549309f5cbd3dda6ec647 P f0d8dd12754e439ca34a41122f43de11: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:33:43.570122 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d4bbac9dd549309f5cbd3dda6ec647" candidate_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0d8dd12754e439ca34a41122f43de11" is_pre_election: true
I20260927 02:33:43.570202 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:43.565985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45769 (local address 127.0.98.3:56947)
0927 02:33:43.566242 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:43.566254 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:43.566330 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:43.566516 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:43.566519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:43.566531 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:33:43.566646 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:43.566652 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.567578 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.567581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:43.568973 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:43.568982 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.569833 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.569837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:43.569933 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:43.569953 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 02:33:43.570022 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 02:33:43.570080 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":112,"threads_started":1}
I20260927 02:33:43.570214 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:43.566060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56947 (local address 127.0.98.2:45769)
0927 02:33:43.566167 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:43.566171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:43.566317 (+ 146us) server_negotiation.cc:407] Connection header received
0927 02:33:43.566401 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:43.566404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:43.566445 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:43.566504 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:33:43.566732 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.567472 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.569082 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.569626 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.569719 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:43.569770 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:43.570050 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 02:33:43.570121 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 02:33:43.570353 2275 raft_consensus.cc:2454] T 28d4bbac9dd549309f5cbd3dda6ec647 P f0d8dd12754e439ca34a41122f43de11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74fd0b61786e4ef2abc5c35685bc05f7 in term 0.
I20260927 02:33:43.570737 2415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d4bbac9dd549309f5cbd3dda6ec647" candidate_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e31b21f7fd594078ae3066bbb49a09c1" is_pre_election: true
I20260927 02:33:43.570766 2486 leader_election.cc:302] T 28d4bbac9dd549309f5cbd3dda6ec647 P 74fd0b61786e4ef2abc5c35685bc05f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74fd0b61786e4ef2abc5c35685bc05f7, f0d8dd12754e439ca34a41122f43de11; no voters:
I20260927 02:33:43.570966 2415 raft_consensus.cc:2454] T 28d4bbac9dd549309f5cbd3dda6ec647 P e31b21f7fd594078ae3066bbb49a09c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74fd0b61786e4ef2abc5c35685bc05f7 in term 0.
I20260927 02:33:43.570981 2632 raft_consensus.cc:2781] T 28d4bbac9dd549309f5cbd3dda6ec647 P 74fd0b61786e4ef2abc5c35685bc05f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:33:43.571049 2632 raft_consensus.cc:482] T 28d4bbac9dd549309f5cbd3dda6ec647 P 74fd0b61786e4ef2abc5c35685bc05f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:33:43.571095 2632 raft_consensus.cc:3037] T 28d4bbac9dd549309f5cbd3dda6ec647 P 74fd0b61786e4ef2abc5c35685bc05f7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:33:43.572057 2632 raft_consensus.cc:504] T 28d4bbac9dd549309f5cbd3dda6ec647 P 74fd0b61786e4ef2abc5c35685bc05f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } } peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } }
I20260927 02:33:43.572307 2632 leader_election.cc:288] T 28d4bbac9dd549309f5cbd3dda6ec647 P 74fd0b61786e4ef2abc5c35685bc05f7 [CANDIDATE]: Term 1 election: Requested vote from peers f0d8dd12754e439ca34a41122f43de11 (127.0.98.1:40513), e31b21f7fd594078ae3066bbb49a09c1 (127.0.98.2:45769)
I20260927 02:33:43.572588 2415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d4bbac9dd549309f5cbd3dda6ec647" candidate_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e31b21f7fd594078ae3066bbb49a09c1"
I20260927 02:33:43.572606 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d4bbac9dd549309f5cbd3dda6ec647" candidate_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0d8dd12754e439ca34a41122f43de11"
I20260927 02:33:43.572698 2415 raft_consensus.cc:3037] T 28d4bbac9dd549309f5cbd3dda6ec647 P e31b21f7fd594078ae3066bbb49a09c1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:33:43.572723 2275 raft_consensus.cc:3037] T 28d4bbac9dd549309f5cbd3dda6ec647 P f0d8dd12754e439ca34a41122f43de11 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:33:43.573866 2275 raft_consensus.cc:2454] T 28d4bbac9dd549309f5cbd3dda6ec647 P f0d8dd12754e439ca34a41122f43de11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74fd0b61786e4ef2abc5c35685bc05f7 in term 1.
I20260927 02:33:43.574119 2486 leader_election.cc:302] T 28d4bbac9dd549309f5cbd3dda6ec647 P 74fd0b61786e4ef2abc5c35685bc05f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74fd0b61786e4ef2abc5c35685bc05f7, f0d8dd12754e439ca34a41122f43de11; no voters:
I20260927 02:33:43.574179 2415 raft_consensus.cc:2454] T 28d4bbac9dd549309f5cbd3dda6ec647 P e31b21f7fd594078ae3066bbb49a09c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74fd0b61786e4ef2abc5c35685bc05f7 in term 1.
I20260927 02:33:43.574286 2632 raft_consensus.cc:2781] T 28d4bbac9dd549309f5cbd3dda6ec647 P 74fd0b61786e4ef2abc5c35685bc05f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:33:43.574460 2632 raft_consensus.cc:686] T 28d4bbac9dd549309f5cbd3dda6ec647 P 74fd0b61786e4ef2abc5c35685bc05f7 [term 1 LEADER]: Becoming Leader. State: Replica: 74fd0b61786e4ef2abc5c35685bc05f7, State: Running, Role: LEADER
I20260927 02:33:43.574759 2632 consensus_queue.cc:237] T 28d4bbac9dd549309f5cbd3dda6ec647 P 74fd0b61786e4ef2abc5c35685bc05f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } } peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } }
I20260927 02:33:43.577472 2117 catalog_manager.cc:5495] T 28d4bbac9dd549309f5cbd3dda6ec647 P 74fd0b61786e4ef2abc5c35685bc05f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74fd0b61786e4ef2abc5c35685bc05f7 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } health_report { overall_health: UNKNOWN } } }
Sep 27 02:33:43 dist-test-slave-grlk krb5kdc[2073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476422, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:33:43.613709 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:43.606471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33328 (local address 127.0.98.62:38251)
0927 02:33:43.606577 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:43.606582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:43.606814 (+ 232us) server_negotiation.cc:407] Connection header received
0927 02:33:43.606928 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:43.606931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:43.606965 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:43.607023 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:43.607630 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.608128 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.608872 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.609120 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.609210 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:43.611212 (+ 2002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:43.611215 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:43.611217 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:43.611237 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:43.612267 (+ 1030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:43.612670 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:43.612672 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:43.612673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:43.612734 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:43.612962 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:43.612965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:43.612967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:43.613131 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:43.613200 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:43.613434 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 02:33:43.613558 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":36}
I20260927 02:33:43.613711 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:43.606433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38251 (local address 127.0.0.1:33328)
0927 02:33:43.606733 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:43.606744 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:43.606847 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:43.607054 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:43.607057 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:43.607434 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:33:43.607524 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:43.607530 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.608219 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.608222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:43.608775 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:43.608782 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.609239 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.609242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:43.609336 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:43.609802 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:33:43.609813 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:33:43.611088 (+ 1275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:33:43.612358 (+ 1270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:43.612366 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:43.612371 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:43.612588 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:43.612796 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:43.612798 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:43.612799 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:43.612870 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:43.613206 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:33:43.613211 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:33:43.613311 (+ 100us) client_negotiation.cc:769] Sending connection context
0927 02:33:43.613396 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 02:33:43.613510 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":102,"threads_started":1}
I20260927 02:33:43.616966 2117 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33328:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 02:33:43.617610 2117 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:33:43.621920 2535 tablet_service.cc:1467] Processing CreateTablet for tablet 5af02d88361d421798fe92eabbcdda86 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=243c7c30cfe2417396ae7e5988ae9493]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:33:43.621903 2395 tablet_service.cc:1467] Processing CreateTablet for tablet 5af02d88361d421798fe92eabbcdda86 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=243c7c30cfe2417396ae7e5988ae9493]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:33:43.621848 2255 tablet_service.cc:1467] Processing CreateTablet for tablet 5af02d88361d421798fe92eabbcdda86 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=243c7c30cfe2417396ae7e5988ae9493]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:33:43.622495 2535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5af02d88361d421798fe92eabbcdda86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:33:43.622501 2395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5af02d88361d421798fe92eabbcdda86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:33:43.622501 2255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5af02d88361d421798fe92eabbcdda86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:33:43.624853 2626 tablet_bootstrap.cc:492] T 5af02d88361d421798fe92eabbcdda86 P 74fd0b61786e4ef2abc5c35685bc05f7: Bootstrap starting.
I20260927 02:33:43.625234 2625 tablet_bootstrap.cc:492] T 5af02d88361d421798fe92eabbcdda86 P e31b21f7fd594078ae3066bbb49a09c1: Bootstrap starting.
I20260927 02:33:43.625818 2626 tablet_bootstrap.cc:654] T 5af02d88361d421798fe92eabbcdda86 P 74fd0b61786e4ef2abc5c35685bc05f7: Neither blocks nor log segments found. Creating new log.
I20260927 02:33:43.626093 2625 tablet_bootstrap.cc:654] T 5af02d88361d421798fe92eabbcdda86 P e31b21f7fd594078ae3066bbb49a09c1: Neither blocks nor log segments found. Creating new log.
I20260927 02:33:43.626638 2624 tablet_bootstrap.cc:492] T 5af02d88361d421798fe92eabbcdda86 P f0d8dd12754e439ca34a41122f43de11: Bootstrap starting.
I20260927 02:33:43.627019 2626 tablet_bootstrap.cc:492] T 5af02d88361d421798fe92eabbcdda86 P 74fd0b61786e4ef2abc5c35685bc05f7: No bootstrap required, opened a new log
I20260927 02:33:43.627105 2626 ts_tablet_manager.cc:1397] T 5af02d88361d421798fe92eabbcdda86 P 74fd0b61786e4ef2abc5c35685bc05f7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:33:43.627161 2625 tablet_bootstrap.cc:492] T 5af02d88361d421798fe92eabbcdda86 P e31b21f7fd594078ae3066bbb49a09c1: No bootstrap required, opened a new log
I20260927 02:33:43.627241 2625 ts_tablet_manager.cc:1397] T 5af02d88361d421798fe92eabbcdda86 P e31b21f7fd594078ae3066bbb49a09c1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:33:43.627486 2624 tablet_bootstrap.cc:654] T 5af02d88361d421798fe92eabbcdda86 P f0d8dd12754e439ca34a41122f43de11: Neither blocks nor log segments found. Creating new log.
I20260927 02:33:43.627523 2626 raft_consensus.cc:348] T 5af02d88361d421798fe92eabbcdda86 P 74fd0b61786e4ef2abc5c35685bc05f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } } peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } }
I20260927 02:33:43.627637 2626 raft_consensus.cc:374] T 5af02d88361d421798fe92eabbcdda86 P 74fd0b61786e4ef2abc5c35685bc05f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:33:43.627678 2626 raft_consensus.cc:729] T 5af02d88361d421798fe92eabbcdda86 P 74fd0b61786e4ef2abc5c35685bc05f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74fd0b61786e4ef2abc5c35685bc05f7, State: Initialized, Role: FOLLOWER
I20260927 02:33:43.627642 2625 raft_consensus.cc:348] T 5af02d88361d421798fe92eabbcdda86 P e31b21f7fd594078ae3066bbb49a09c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } } peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } }
I20260927 02:33:43.627743 2625 raft_consensus.cc:374] T 5af02d88361d421798fe92eabbcdda86 P e31b21f7fd594078ae3066bbb49a09c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:33:43.627792 2625 raft_consensus.cc:729] T 5af02d88361d421798fe92eabbcdda86 P e31b21f7fd594078ae3066bbb49a09c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e31b21f7fd594078ae3066bbb49a09c1, State: Initialized, Role: FOLLOWER
I20260927 02:33:43.627801 2626 consensus_queue.cc:260] T 5af02d88361d421798fe92eabbcdda86 P 74fd0b61786e4ef2abc5c35685bc05f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } } peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } }
I20260927 02:33:43.627918 2625 consensus_queue.cc:260] T 5af02d88361d421798fe92eabbcdda86 P e31b21f7fd594078ae3066bbb49a09c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } } peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } }
I20260927 02:33:43.628038 2626 ts_tablet_manager.cc:1428] T 5af02d88361d421798fe92eabbcdda86 P 74fd0b61786e4ef2abc5c35685bc05f7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:33:43.628172 2625 ts_tablet_manager.cc:1428] T 5af02d88361d421798fe92eabbcdda86 P e31b21f7fd594078ae3066bbb49a09c1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:33:43.628847 2632 tablet_replica.cc:439] T 5af02d88361d421798fe92eabbcdda86 P 74fd0b61786e4ef2abc5c35685bc05f7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } } peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } } }
I20260927 02:33:43.628980 2632 tablet_replica.cc:442] T 5af02d88361d421798fe92eabbcdda86 P 74fd0b61786e4ef2abc5c35685bc05f7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:33:43.628182 2631 tablet_replica.cc:439] T 5af02d88361d421798fe92eabbcdda86 P e31b21f7fd594078ae3066bbb49a09c1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } } peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } } }
I20260927 02:33:43.629180 2631 tablet_replica.cc:442] T 5af02d88361d421798fe92eabbcdda86 P e31b21f7fd594078ae3066bbb49a09c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:33:43.629606 2624 tablet_bootstrap.cc:492] T 5af02d88361d421798fe92eabbcdda86 P f0d8dd12754e439ca34a41122f43de11: No bootstrap required, opened a new log
I20260927 02:33:43.629683 2624 ts_tablet_manager.cc:1397] T 5af02d88361d421798fe92eabbcdda86 P f0d8dd12754e439ca34a41122f43de11: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260927 02:33:43.630077 2624 raft_consensus.cc:348] T 5af02d88361d421798fe92eabbcdda86 P f0d8dd12754e439ca34a41122f43de11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } } peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } }
I20260927 02:33:43.630192 2624 raft_consensus.cc:374] T 5af02d88361d421798fe92eabbcdda86 P f0d8dd12754e439ca34a41122f43de11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:33:43.630234 2624 raft_consensus.cc:729] T 5af02d88361d421798fe92eabbcdda86 P f0d8dd12754e439ca34a41122f43de11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0d8dd12754e439ca34a41122f43de11, State: Initialized, Role: FOLLOWER
I20260927 02:33:43.630347 2624 consensus_queue.cc:260] T 5af02d88361d421798fe92eabbcdda86 P f0d8dd12754e439ca34a41122f43de11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } } peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } }
I20260927 02:33:43.630589 2624 ts_tablet_manager.cc:1428] T 5af02d88361d421798fe92eabbcdda86 P f0d8dd12754e439ca34a41122f43de11: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 02:33:43.630600 2630 tablet_replica.cc:439] T 5af02d88361d421798fe92eabbcdda86 P f0d8dd12754e439ca34a41122f43de11: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } } peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } } }
I20260927 02:33:43.630751 2630 tablet_replica.cc:442] T 5af02d88361d421798fe92eabbcdda86 P f0d8dd12754e439ca34a41122f43de11: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:33:43.686302 2631 raft_consensus.cc:482] T 5af02d88361d421798fe92eabbcdda86 P e31b21f7fd594078ae3066bbb49a09c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:33:43.686435 2631 raft_consensus.cc:504] T 5af02d88361d421798fe92eabbcdda86 P e31b21f7fd594078ae3066bbb49a09c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } } peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } }
I20260927 02:33:43.687458 2631 leader_election.cc:288] T 5af02d88361d421798fe92eabbcdda86 P e31b21f7fd594078ae3066bbb49a09c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0d8dd12754e439ca34a41122f43de11 (127.0.98.1:40513), 74fd0b61786e4ef2abc5c35685bc05f7 (127.0.98.3:34811)
I20260927 02:33:43.691339 2354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:43.687642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40513 (local address 127.0.98.2:39819)
0927 02:33:43.687805 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:43.687822 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:43.687958 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:43.688237 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:43.688240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:43.688260 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:33:43.688393 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:43.688400 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.689219 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.689222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:43.690077 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:43.690087 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.690794 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.690798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:43.691027 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:43.691057 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 02:33:43.691150 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 02:33:43.691226 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 02:33:43.691355 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:43.687835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39819 (local address 127.0.98.1:40513)
0927 02:33:43.687940 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:43.687944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:43.687959 (+ 15us) server_negotiation.cc:407] Connection header received
0927 02:33:43.688075 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:43.688078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:43.688129 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:43.688215 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:33:43.688508 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.689119 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.690199 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.690650 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.690737 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:43.690809 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:43.691162 (+ 353us) server_negotiation.cc:299] Negotiation successful
0927 02:33:43.691232 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 02:33:43.691532 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:43.687732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34811 (local address 127.0.98.2:53883)
0927 02:33:43.688140 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:43.688153 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:43.688240 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:43.688509 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:43.688511 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:43.688526 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:33:43.688640 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:43.688646 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.689445 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.689448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:43.690274 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:43.690282 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.691157 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.691160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:43.691259 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:43.691279 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 02:33:43.691353 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 02:33:43.691413 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":120,"threads_started":1}
I20260927 02:33:43.691603 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:43.687827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53883 (local address 127.0.98.3:34811)
0927 02:33:43.687936 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:43.687940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:43.688233 (+ 293us) server_negotiation.cc:407] Connection header received
0927 02:33:43.688339 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:43.688342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:43.688415 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:43.688486 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:33:43.688742 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.689330 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.690399 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.691019 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.691136 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:43.691198 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:43.691387 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 02:33:43.691462 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 02:33:43.691951 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5af02d88361d421798fe92eabbcdda86" candidate_uuid: "e31b21f7fd594078ae3066bbb49a09c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0d8dd12754e439ca34a41122f43de11" is_pre_election: true
I20260927 02:33:43.692097 2275 raft_consensus.cc:2454] T 5af02d88361d421798fe92eabbcdda86 P f0d8dd12754e439ca34a41122f43de11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e31b21f7fd594078ae3066bbb49a09c1 in term 0.
I20260927 02:33:43.692051 2555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5af02d88361d421798fe92eabbcdda86" candidate_uuid: "e31b21f7fd594078ae3066bbb49a09c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" is_pre_election: true
I20260927 02:33:43.692267 2555 raft_consensus.cc:2454] T 5af02d88361d421798fe92eabbcdda86 P 74fd0b61786e4ef2abc5c35685bc05f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e31b21f7fd594078ae3066bbb49a09c1 in term 0.
I20260927 02:33:43.692478 2346 leader_election.cc:302] T 5af02d88361d421798fe92eabbcdda86 P e31b21f7fd594078ae3066bbb49a09c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e31b21f7fd594078ae3066bbb49a09c1, f0d8dd12754e439ca34a41122f43de11; no voters:
I20260927 02:33:43.692740 2631 raft_consensus.cc:2781] T 5af02d88361d421798fe92eabbcdda86 P e31b21f7fd594078ae3066bbb49a09c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:33:43.692796 2631 raft_consensus.cc:482] T 5af02d88361d421798fe92eabbcdda86 P e31b21f7fd594078ae3066bbb49a09c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:33:43.692835 2631 raft_consensus.cc:3037] T 5af02d88361d421798fe92eabbcdda86 P e31b21f7fd594078ae3066bbb49a09c1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:33:43.693799 2631 raft_consensus.cc:504] T 5af02d88361d421798fe92eabbcdda86 P e31b21f7fd594078ae3066bbb49a09c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } } peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } }
I20260927 02:33:43.694104 2631 leader_election.cc:288] T 5af02d88361d421798fe92eabbcdda86 P e31b21f7fd594078ae3066bbb49a09c1 [CANDIDATE]: Term 1 election: Requested vote from peers f0d8dd12754e439ca34a41122f43de11 (127.0.98.1:40513), 74fd0b61786e4ef2abc5c35685bc05f7 (127.0.98.3:34811)
I20260927 02:33:43.694433 2555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5af02d88361d421798fe92eabbcdda86" candidate_uuid: "e31b21f7fd594078ae3066bbb49a09c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74fd0b61786e4ef2abc5c35685bc05f7"
I20260927 02:33:43.694438 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5af02d88361d421798fe92eabbcdda86" candidate_uuid: "e31b21f7fd594078ae3066bbb49a09c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0d8dd12754e439ca34a41122f43de11"
I20260927 02:33:43.694545 2555 raft_consensus.cc:3037] T 5af02d88361d421798fe92eabbcdda86 P 74fd0b61786e4ef2abc5c35685bc05f7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:33:43.694546 2275 raft_consensus.cc:3037] T 5af02d88361d421798fe92eabbcdda86 P f0d8dd12754e439ca34a41122f43de11 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:33:43.695590 2275 raft_consensus.cc:2454] T 5af02d88361d421798fe92eabbcdda86 P f0d8dd12754e439ca34a41122f43de11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e31b21f7fd594078ae3066bbb49a09c1 in term 1.
I20260927 02:33:43.695590 2555 raft_consensus.cc:2454] T 5af02d88361d421798fe92eabbcdda86 P 74fd0b61786e4ef2abc5c35685bc05f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e31b21f7fd594078ae3066bbb49a09c1 in term 1.
I20260927 02:33:43.695850 2346 leader_election.cc:302] T 5af02d88361d421798fe92eabbcdda86 P e31b21f7fd594078ae3066bbb49a09c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e31b21f7fd594078ae3066bbb49a09c1, f0d8dd12754e439ca34a41122f43de11; no voters:
I20260927 02:33:43.696003 2631 raft_consensus.cc:2781] T 5af02d88361d421798fe92eabbcdda86 P e31b21f7fd594078ae3066bbb49a09c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:33:43.696197 2631 raft_consensus.cc:686] T 5af02d88361d421798fe92eabbcdda86 P e31b21f7fd594078ae3066bbb49a09c1 [term 1 LEADER]: Becoming Leader. State: Replica: e31b21f7fd594078ae3066bbb49a09c1, State: Running, Role: LEADER
I20260927 02:33:43.696491 2631 consensus_queue.cc:237] T 5af02d88361d421798fe92eabbcdda86 P e31b21f7fd594078ae3066bbb49a09c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } } peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } }
I20260927 02:33:43.698657 2646 tablet_replica.cc:439] T 5af02d88361d421798fe92eabbcdda86 P e31b21f7fd594078ae3066bbb49a09c1: TxnStatusTablet state changed. Reason: New leader e31b21f7fd594078ae3066bbb49a09c1. Latest consensus state: current_term: 1 leader_uuid: "e31b21f7fd594078ae3066bbb49a09c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } } peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } } }
I20260927 02:33:43.698781 2646 tablet_replica.cc:442] T 5af02d88361d421798fe92eabbcdda86 P e31b21f7fd594078ae3066bbb49a09c1: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:33:43.699242 2648 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 02:33:43.699412 2648 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 02:33:43.699627 2117 catalog_manager.cc:5495] T 5af02d88361d421798fe92eabbcdda86 P e31b21f7fd594078ae3066bbb49a09c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to e31b21f7fd594078ae3066bbb49a09c1 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "e31b21f7fd594078ae3066bbb49a09c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } health_report { overall_health: UNKNOWN } } }
W20260927 02:33:43.720697 2602 tablet.cc:2369] T 28d4bbac9dd549309f5cbd3dda6ec647 P 74fd0b61786e4ef2abc5c35685bc05f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:33:43.729323 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:43.726047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53556 (local address 127.0.98.2:45769)
0927 02:33:43.726189 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:43.726194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:43.726207 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:33:43.726275 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:43.726278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:43.726327 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:43.726403 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:33:43.726626 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.727131 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.727958 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.728174 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.728255 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:43.728518 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:33:43.728922 (+ 404us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:33:43.729048 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:43.729122 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 02:33:43.729172 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 02:33:43.729305 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:43.725907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45769 (local address 127.0.0.1:53556)
0927 02:33:43.726045 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:43.726061 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:43.726162 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:43.726423 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:43.726426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:43.726433 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:33:43.726531 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:43.726536 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.727230 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.727234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:43.727858 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:43.727866 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.728264 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.728268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:43.728354 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:43.728393 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:33:43.729033 (+ 640us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:33:43.729047 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 02:33:43.729112 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 02:33:43.729192 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 02:33:43.733942 2555 raft_consensus.cc:1261] T 5af02d88361d421798fe92eabbcdda86 P 74fd0b61786e4ef2abc5c35685bc05f7 [term 1 FOLLOWER]: Refusing update from remote peer e31b21f7fd594078ae3066bbb49a09c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:33:43.734009 2275 raft_consensus.cc:1261] T 5af02d88361d421798fe92eabbcdda86 P f0d8dd12754e439ca34a41122f43de11 [term 1 FOLLOWER]: Refusing update from remote peer e31b21f7fd594078ae3066bbb49a09c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:33:43.734594 2646 consensus_queue.cc:1035] T 5af02d88361d421798fe92eabbcdda86 P e31b21f7fd594078ae3066bbb49a09c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:33:43.734776 2631 consensus_queue.cc:1035] T 5af02d88361d421798fe92eabbcdda86 P e31b21f7fd594078ae3066bbb49a09c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:33:43.737860 2632 tablet_replica.cc:439] T 5af02d88361d421798fe92eabbcdda86 P 74fd0b61786e4ef2abc5c35685bc05f7: TxnStatusTablet state changed. Reason: New leader e31b21f7fd594078ae3066bbb49a09c1. Latest consensus state: current_term: 1 leader_uuid: "e31b21f7fd594078ae3066bbb49a09c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } } peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } } }
I20260927 02:33:43.737982 2632 tablet_replica.cc:442] T 5af02d88361d421798fe92eabbcdda86 P 74fd0b61786e4ef2abc5c35685bc05f7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:33:43.738238 2630 tablet_replica.cc:439] T 5af02d88361d421798fe92eabbcdda86 P f0d8dd12754e439ca34a41122f43de11: TxnStatusTablet state changed. Reason: New leader e31b21f7fd594078ae3066bbb49a09c1. Latest consensus state: current_term: 1 leader_uuid: "e31b21f7fd594078ae3066bbb49a09c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } } peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } } }
I20260927 02:33:43.738341 2630 tablet_replica.cc:442] T 5af02d88361d421798fe92eabbcdda86 P f0d8dd12754e439ca34a41122f43de11: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:33:43.741181 2631 tablet_replica.cc:439] T 5af02d88361d421798fe92eabbcdda86 P e31b21f7fd594078ae3066bbb49a09c1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e31b21f7fd594078ae3066bbb49a09c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } } peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } } }
I20260927 02:33:43.741298 2631 tablet_replica.cc:442] T 5af02d88361d421798fe92eabbcdda86 P e31b21f7fd594078ae3066bbb49a09c1: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:33:43.741578 2630 tablet_replica.cc:439] T 5af02d88361d421798fe92eabbcdda86 P f0d8dd12754e439ca34a41122f43de11: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e31b21f7fd594078ae3066bbb49a09c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } } peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } } }
I20260927 02:33:43.741695 2630 tablet_replica.cc:442] T 5af02d88361d421798fe92eabbcdda86 P f0d8dd12754e439ca34a41122f43de11: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:33:43.742257 2652 mvcc.cc:205] Tried to move back new op lower bound from 7333791431607848960 to 7333791431463391232. Current Snapshot: MvccSnapshot[applied={T|T < 7333791431607848960}]
I20260927 02:33:43.743469 2646 tablet_replica.cc:439] T 5af02d88361d421798fe92eabbcdda86 P e31b21f7fd594078ae3066bbb49a09c1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e31b21f7fd594078ae3066bbb49a09c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } } peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } } }
I20260927 02:33:43.743587 2646 tablet_replica.cc:442] T 5af02d88361d421798fe92eabbcdda86 P e31b21f7fd594078ae3066bbb49a09c1: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:33:43.744518 2653 mvcc.cc:205] Tried to move back new op lower bound from 7333791431607848960 to 7333791431463391232. Current Snapshot: MvccSnapshot[applied={T|T < 7333791431607848960}]
I20260927 02:33:43.744711 2632 tablet_replica.cc:439] T 5af02d88361d421798fe92eabbcdda86 P 74fd0b61786e4ef2abc5c35685bc05f7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e31b21f7fd594078ae3066bbb49a09c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } } peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } } }
I20260927 02:33:43.744819 2632 tablet_replica.cc:442] T 5af02d88361d421798fe92eabbcdda86 P 74fd0b61786e4ef2abc5c35685bc05f7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:33:43.752888 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:43.749276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44992 (local address 127.0.98.3:34811)
0927 02:33:43.749402 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:43.749406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:43.749423 (+ 17us) server_negotiation.cc:407] Connection header received
0927 02:33:43.749495 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:43.749498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:43.749557 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:43.749658 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:33:43.749874 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.750655 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.751451 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.751794 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.751901 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:43.752085 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:33:43.752546 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:33:43.752660 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:43.752695 (+ 35us) server_negotiation.cc:299] Negotiation successful
0927 02:33:43.752752 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 02:33:43.762948 2607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:43.759561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45769 (local address 127.0.98.3:33203)
0927 02:33:43.759703 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:43.759717 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:43.759786 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:43.760051 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:43.760053 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:43.760059 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:33:43.760146 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:43.760150 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.761133 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.761136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:43.761742 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:43.761750 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.762293 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.762295 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:43.762364 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:43.762404 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:33:43.762699 (+ 295us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:33:43.762704 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:33:43.762750 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 02:33:43.762812 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 02:33:43.762970 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:43.759677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33203 (local address 127.0.98.2:45769)
0927 02:33:43.759793 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:43.759797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:43.759841 (+ 44us) server_negotiation.cc:407] Connection header received
0927 02:33:43.759892 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:43.759896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:43.759962 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:43.760039 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:33:43.760234 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.761024 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.761846 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.762176 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.762282 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:43.762537 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:33:43.762616 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:33:43.762689 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:43.762779 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 02:33:43.762833 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 02:33:43.769219 2275 raft_consensus.cc:1261] T 28d4bbac9dd549309f5cbd3dda6ec647 P f0d8dd12754e439ca34a41122f43de11 [term 1 FOLLOWER]: Refusing update from remote peer 74fd0b61786e4ef2abc5c35685bc05f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:33:43.769476 2415 raft_consensus.cc:1261] T 28d4bbac9dd549309f5cbd3dda6ec647 P e31b21f7fd594078ae3066bbb49a09c1 [term 1 FOLLOWER]: Refusing update from remote peer 74fd0b61786e4ef2abc5c35685bc05f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:33:43.769662 2634 consensus_queue.cc:1035] T 28d4bbac9dd549309f5cbd3dda6ec647 P 74fd0b61786e4ef2abc5c35685bc05f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:33:43.769970 2632 consensus_queue.cc:1035] T 28d4bbac9dd549309f5cbd3dda6ec647 P 74fd0b61786e4ef2abc5c35685bc05f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 02:33:43.771903 2462 tablet.cc:2369] T 28d4bbac9dd549309f5cbd3dda6ec647 P e31b21f7fd594078ae3066bbb49a09c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:33:43.777005 2651 mvcc.cc:205] Tried to move back new op lower bound from 7333791431754723328 to 7333791430964334592. Current Snapshot: MvccSnapshot[applied={T|T < 7333791431754723328}]
I20260927 02:33:43.793301 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:43.788909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34811 (local address 127.0.98.2:48523)
0927 02:33:43.789167 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:43.789179 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:43.789254 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:43.789577 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:43.789580 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:43.789589 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:33:43.789689 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:43.789696 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.790824 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.790827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:43.791783 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:43.791793 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.792535 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.792538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:43.792633 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:43.792682 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:33:43.793015 (+ 333us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:33:43.793021 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 02:33:43.793079 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 02:33:43.793159 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":82,"threads_started":1}
I20260927 02:33:43.793287 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:43.788982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48523 (local address 127.0.98.3:34811)
0927 02:33:43.789204 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:43.789209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:43.789226 (+ 17us) server_negotiation.cc:407] Connection header received
0927 02:33:43.789327 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:43.789330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:43.789389 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:43.789466 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:33:43.789807 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.790657 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.791891 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.792137 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.792231 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:43.792817 (+ 586us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:33:43.792899 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:33:43.792997 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:43.793088 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 02:33:43.793159 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260927 02:33:43.817036 2118 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:33:43.817211 2118 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:33:43.819463 2118 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 307e7dee3cca4210b78b3e0eeba083a4: Sending DeleteTablet for 3 replicas of tablet 28d4bbac9dd549309f5cbd3dda6ec647
I20260927 02:33:43.820179 2395 tablet_service.cc:1514] Processing DeleteTablet for tablet 28d4bbac9dd549309f5cbd3dda6ec647 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:33:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53548
I20260927 02:33:43.820309 2255 tablet_service.cc:1514] Processing DeleteTablet for tablet 28d4bbac9dd549309f5cbd3dda6ec647 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:33:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43464
I20260927 02:33:43.821081 2535 tablet_service.cc:1514] Processing DeleteTablet for tablet 28d4bbac9dd549309f5cbd3dda6ec647 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:33:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44980
I20260927 02:33:43.821334 2118 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33316:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:33:43.821702 2118 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:33:43.826211 2395 tablet_service.cc:1467] Processing CreateTablet for tablet 98fadf4cc5fa46c9958d18ddf54ece4e (DEFAULT_TABLE table=test-table [id=1930df9498b74587b8c72e6928a12c06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:33:43.826493 2395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98fadf4cc5fa46c9958d18ddf54ece4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:33:43.826560 2255 tablet_service.cc:1467] Processing CreateTablet for tablet 98fadf4cc5fa46c9958d18ddf54ece4e (DEFAULT_TABLE table=test-table [id=1930df9498b74587b8c72e6928a12c06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:33:43.826787 2255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98fadf4cc5fa46c9958d18ddf54ece4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:33:43.829098 2535 tablet_service.cc:1467] Processing CreateTablet for tablet 98fadf4cc5fa46c9958d18ddf54ece4e (DEFAULT_TABLE table=test-table [id=1930df9498b74587b8c72e6928a12c06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:33:43.829258 2625 tablet_bootstrap.cc:492] T 98fadf4cc5fa46c9958d18ddf54ece4e P e31b21f7fd594078ae3066bbb49a09c1: Bootstrap starting.
I20260927 02:33:43.829372 2535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98fadf4cc5fa46c9958d18ddf54ece4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:33:43.830456 2625 tablet_bootstrap.cc:654] T 98fadf4cc5fa46c9958d18ddf54ece4e P e31b21f7fd594078ae3066bbb49a09c1: Neither blocks nor log segments found. Creating new log.
I20260927 02:33:43.831017 2624 tablet_bootstrap.cc:492] T 98fadf4cc5fa46c9958d18ddf54ece4e P f0d8dd12754e439ca34a41122f43de11: Bootstrap starting.
I20260927 02:33:43.831621 2626 tablet_bootstrap.cc:492] T 98fadf4cc5fa46c9958d18ddf54ece4e P 74fd0b61786e4ef2abc5c35685bc05f7: Bootstrap starting.
I20260927 02:33:43.831918 2624 tablet_bootstrap.cc:654] T 98fadf4cc5fa46c9958d18ddf54ece4e P f0d8dd12754e439ca34a41122f43de11: Neither blocks nor log segments found. Creating new log.
I20260927 02:33:43.832456 2626 tablet_bootstrap.cc:654] T 98fadf4cc5fa46c9958d18ddf54ece4e P 74fd0b61786e4ef2abc5c35685bc05f7: Neither blocks nor log segments found. Creating new log.
I20260927 02:33:43.829104 2672 tablet_replica.cc:330] T 28d4bbac9dd549309f5cbd3dda6ec647 P 74fd0b61786e4ef2abc5c35685bc05f7: stopping tablet replica
I20260927 02:33:43.833330 2624 tablet_bootstrap.cc:492] T 98fadf4cc5fa46c9958d18ddf54ece4e P f0d8dd12754e439ca34a41122f43de11: No bootstrap required, opened a new log
I20260927 02:33:43.833390 2624 ts_tablet_manager.cc:1397] T 98fadf4cc5fa46c9958d18ddf54ece4e P f0d8dd12754e439ca34a41122f43de11: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 02:33:43.833571 2626 tablet_bootstrap.cc:492] T 98fadf4cc5fa46c9958d18ddf54ece4e P 74fd0b61786e4ef2abc5c35685bc05f7: No bootstrap required, opened a new log
I20260927 02:33:43.833662 2626 ts_tablet_manager.cc:1397] T 98fadf4cc5fa46c9958d18ddf54ece4e P 74fd0b61786e4ef2abc5c35685bc05f7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 02:33:43.833845 2624 raft_consensus.cc:348] T 98fadf4cc5fa46c9958d18ddf54ece4e P f0d8dd12754e439ca34a41122f43de11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } } peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } }
I20260927 02:33:43.833956 2624 raft_consensus.cc:374] T 98fadf4cc5fa46c9958d18ddf54ece4e P f0d8dd12754e439ca34a41122f43de11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:33:43.833987 2624 raft_consensus.cc:729] T 98fadf4cc5fa46c9958d18ddf54ece4e P f0d8dd12754e439ca34a41122f43de11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0d8dd12754e439ca34a41122f43de11, State: Initialized, Role: FOLLOWER
I20260927 02:33:43.834064 2626 raft_consensus.cc:348] T 98fadf4cc5fa46c9958d18ddf54ece4e P 74fd0b61786e4ef2abc5c35685bc05f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } } peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } }
I20260927 02:33:43.834100 2624 consensus_queue.cc:260] T 98fadf4cc5fa46c9958d18ddf54ece4e P f0d8dd12754e439ca34a41122f43de11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } } peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } }
I20260927 02:33:43.834180 2626 raft_consensus.cc:374] T 98fadf4cc5fa46c9958d18ddf54ece4e P 74fd0b61786e4ef2abc5c35685bc05f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:33:43.834211 2626 raft_consensus.cc:729] T 98fadf4cc5fa46c9958d18ddf54ece4e P 74fd0b61786e4ef2abc5c35685bc05f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74fd0b61786e4ef2abc5c35685bc05f7, State: Initialized, Role: FOLLOWER
I20260927 02:33:43.834312 2626 consensus_queue.cc:260] T 98fadf4cc5fa46c9958d18ddf54ece4e P 74fd0b61786e4ef2abc5c35685bc05f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } } peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } }
I20260927 02:33:43.834374 2624 ts_tablet_manager.cc:1428] T 98fadf4cc5fa46c9958d18ddf54ece4e P f0d8dd12754e439ca34a41122f43de11: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 02:33:43.834534 2626 ts_tablet_manager.cc:1428] T 98fadf4cc5fa46c9958d18ddf54ece4e P 74fd0b61786e4ef2abc5c35685bc05f7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:33:43.834728 2672 raft_consensus.cc:2229] T 28d4bbac9dd549309f5cbd3dda6ec647 P 74fd0b61786e4ef2abc5c35685bc05f7 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:33:43.835071 2672 raft_consensus.cc:2258] T 28d4bbac9dd549309f5cbd3dda6ec647 P 74fd0b61786e4ef2abc5c35685bc05f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:33:43.836254 2625 tablet_bootstrap.cc:492] T 98fadf4cc5fa46c9958d18ddf54ece4e P e31b21f7fd594078ae3066bbb49a09c1: No bootstrap required, opened a new log
I20260927 02:33:43.836344 2625 ts_tablet_manager.cc:1397] T 98fadf4cc5fa46c9958d18ddf54ece4e P e31b21f7fd594078ae3066bbb49a09c1: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260927 02:33:43.836668 2672 ts_tablet_manager.cc:1905] T 28d4bbac9dd549309f5cbd3dda6ec647 P 74fd0b61786e4ef2abc5c35685bc05f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:33:43.836776 2625 raft_consensus.cc:348] T 98fadf4cc5fa46c9958d18ddf54ece4e P e31b21f7fd594078ae3066bbb49a09c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } } peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } }
I20260927 02:33:43.836889 2625 raft_consensus.cc:374] T 98fadf4cc5fa46c9958d18ddf54ece4e P e31b21f7fd594078ae3066bbb49a09c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:33:43.836921 2625 raft_consensus.cc:729] T 98fadf4cc5fa46c9958d18ddf54ece4e P e31b21f7fd594078ae3066bbb49a09c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e31b21f7fd594078ae3066bbb49a09c1, State: Initialized, Role: FOLLOWER
I20260927 02:33:43.837061 2625 consensus_queue.cc:260] T 98fadf4cc5fa46c9958d18ddf54ece4e P e31b21f7fd594078ae3066bbb49a09c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } } peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } }
I20260927 02:33:43.838304 2625 ts_tablet_manager.cc:1428] T 98fadf4cc5fa46c9958d18ddf54ece4e P e31b21f7fd594078ae3066bbb49a09c1: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260927 02:33:43.839020 2670 tablet_replica.cc:330] T 28d4bbac9dd549309f5cbd3dda6ec647 P e31b21f7fd594078ae3066bbb49a09c1: stopping tablet replica
I20260927 02:33:43.839370 2672 ts_tablet_manager.cc:1918] T 28d4bbac9dd549309f5cbd3dda6ec647 P 74fd0b61786e4ef2abc5c35685bc05f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:33:43.839457 2672 log.cc:1196] T 28d4bbac9dd549309f5cbd3dda6ec647 P 74fd0b61786e4ef2abc5c35685bc05f7: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-2/wal/wals/28d4bbac9dd549309f5cbd3dda6ec647
I20260927 02:33:43.839743 2672 ts_tablet_manager.cc:1939] T 28d4bbac9dd549309f5cbd3dda6ec647 P 74fd0b61786e4ef2abc5c35685bc05f7: Deleting consensus metadata
I20260927 02:33:43.840829 2103 catalog_manager.cc:4841] TS 74fd0b61786e4ef2abc5c35685bc05f7 (127.0.98.3:34811): tablet 28d4bbac9dd549309f5cbd3dda6ec647 (table test-table [id=a8278cd8a3f44f8d86f9bd0b4c683606]) successfully deleted
I20260927 02:33:43.841940 2670 raft_consensus.cc:2229] T 28d4bbac9dd549309f5cbd3dda6ec647 P e31b21f7fd594078ae3066bbb49a09c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:33:43.842152 2670 raft_consensus.cc:2258] T 28d4bbac9dd549309f5cbd3dda6ec647 P e31b21f7fd594078ae3066bbb49a09c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:33:43.843385 2670 ts_tablet_manager.cc:1905] T 28d4bbac9dd549309f5cbd3dda6ec647 P e31b21f7fd594078ae3066bbb49a09c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:33:43.844831 2671 tablet_replica.cc:330] T 28d4bbac9dd549309f5cbd3dda6ec647 P f0d8dd12754e439ca34a41122f43de11: stopping tablet replica
I20260927 02:33:43.845167 2671 raft_consensus.cc:2229] T 28d4bbac9dd549309f5cbd3dda6ec647 P f0d8dd12754e439ca34a41122f43de11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:33:43.845408 2671 raft_consensus.cc:2258] T 28d4bbac9dd549309f5cbd3dda6ec647 P f0d8dd12754e439ca34a41122f43de11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:33:43.850044 2670 ts_tablet_manager.cc:1918] T 28d4bbac9dd549309f5cbd3dda6ec647 P e31b21f7fd594078ae3066bbb49a09c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:33:43.850147 2670 log.cc:1196] T 28d4bbac9dd549309f5cbd3dda6ec647 P e31b21f7fd594078ae3066bbb49a09c1: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-1/wal/wals/28d4bbac9dd549309f5cbd3dda6ec647
I20260927 02:33:43.850425 2670 ts_tablet_manager.cc:1939] T 28d4bbac9dd549309f5cbd3dda6ec647 P e31b21f7fd594078ae3066bbb49a09c1: Deleting consensus metadata
I20260927 02:33:43.851447 2105 catalog_manager.cc:4841] TS e31b21f7fd594078ae3066bbb49a09c1 (127.0.98.2:45769): tablet 28d4bbac9dd549309f5cbd3dda6ec647 (table test-table [id=a8278cd8a3f44f8d86f9bd0b4c683606]) successfully deleted
I20260927 02:33:43.852181 2671 ts_tablet_manager.cc:1905] T 28d4bbac9dd549309f5cbd3dda6ec647 P f0d8dd12754e439ca34a41122f43de11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:33:43.854424 2671 ts_tablet_manager.cc:1918] T 28d4bbac9dd549309f5cbd3dda6ec647 P f0d8dd12754e439ca34a41122f43de11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:33:43.854507 2671 log.cc:1196] T 28d4bbac9dd549309f5cbd3dda6ec647 P f0d8dd12754e439ca34a41122f43de11: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-0/wal/wals/28d4bbac9dd549309f5cbd3dda6ec647
I20260927 02:33:43.854740 2671 ts_tablet_manager.cc:1939] T 28d4bbac9dd549309f5cbd3dda6ec647 P f0d8dd12754e439ca34a41122f43de11: Deleting consensus metadata
I20260927 02:33:43.855870 2102 catalog_manager.cc:4841] TS f0d8dd12754e439ca34a41122f43de11 (127.0.98.1:40513): tablet 28d4bbac9dd549309f5cbd3dda6ec647 (table test-table [id=a8278cd8a3f44f8d86f9bd0b4c683606]) successfully deleted
I20260927 02:33:43.891624 2646 raft_consensus.cc:482] T 98fadf4cc5fa46c9958d18ddf54ece4e P e31b21f7fd594078ae3066bbb49a09c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:33:43.891762 2646 raft_consensus.cc:504] T 98fadf4cc5fa46c9958d18ddf54ece4e P e31b21f7fd594078ae3066bbb49a09c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } } peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } }
I20260927 02:33:43.891994 2630 raft_consensus.cc:482] T 98fadf4cc5fa46c9958d18ddf54ece4e P f0d8dd12754e439ca34a41122f43de11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:33:43.892091 2630 raft_consensus.cc:504] T 98fadf4cc5fa46c9958d18ddf54ece4e P f0d8dd12754e439ca34a41122f43de11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } } peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } }
I20260927 02:33:43.892223 2646 leader_election.cc:288] T 98fadf4cc5fa46c9958d18ddf54ece4e P e31b21f7fd594078ae3066bbb49a09c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0d8dd12754e439ca34a41122f43de11 (127.0.98.1:40513), 74fd0b61786e4ef2abc5c35685bc05f7 (127.0.98.3:34811)
I20260927 02:33:43.892824 2555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fadf4cc5fa46c9958d18ddf54ece4e" candidate_uuid: "e31b21f7fd594078ae3066bbb49a09c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" is_pre_election: true
I20260927 02:33:43.892980 2555 raft_consensus.cc:2454] T 98fadf4cc5fa46c9958d18ddf54ece4e P 74fd0b61786e4ef2abc5c35685bc05f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e31b21f7fd594078ae3066bbb49a09c1 in term 0.
I20260927 02:33:43.893293 2347 leader_election.cc:302] T 98fadf4cc5fa46c9958d18ddf54ece4e P e31b21f7fd594078ae3066bbb49a09c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74fd0b61786e4ef2abc5c35685bc05f7, e31b21f7fd594078ae3066bbb49a09c1; no voters:
I20260927 02:33:43.893311 2630 leader_election.cc:288] T 98fadf4cc5fa46c9958d18ddf54ece4e P f0d8dd12754e439ca34a41122f43de11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e31b21f7fd594078ae3066bbb49a09c1 (127.0.98.2:45769), 74fd0b61786e4ef2abc5c35685bc05f7 (127.0.98.3:34811)
I20260927 02:33:43.893498 2646 raft_consensus.cc:2781] T 98fadf4cc5fa46c9958d18ddf54ece4e P e31b21f7fd594078ae3066bbb49a09c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:33:43.893545 2646 raft_consensus.cc:482] T 98fadf4cc5fa46c9958d18ddf54ece4e P e31b21f7fd594078ae3066bbb49a09c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:33:43.893579 2646 raft_consensus.cc:3037] T 98fadf4cc5fa46c9958d18ddf54ece4e P e31b21f7fd594078ae3066bbb49a09c1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:33:43.894773 2646 raft_consensus.cc:504] T 98fadf4cc5fa46c9958d18ddf54ece4e P e31b21f7fd594078ae3066bbb49a09c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } } peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } }
I20260927 02:33:43.895079 2646 leader_election.cc:288] T 98fadf4cc5fa46c9958d18ddf54ece4e P e31b21f7fd594078ae3066bbb49a09c1 [CANDIDATE]: Term 1 election: Requested vote from peers f0d8dd12754e439ca34a41122f43de11 (127.0.98.1:40513), 74fd0b61786e4ef2abc5c35685bc05f7 (127.0.98.3:34811)
I20260927 02:33:43.895054 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fadf4cc5fa46c9958d18ddf54ece4e" candidate_uuid: "e31b21f7fd594078ae3066bbb49a09c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0d8dd12754e439ca34a41122f43de11" is_pre_election: true
I20260927 02:33:43.895179 2275 raft_consensus.cc:2454] T 98fadf4cc5fa46c9958d18ddf54ece4e P f0d8dd12754e439ca34a41122f43de11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e31b21f7fd594078ae3066bbb49a09c1 in term 0.
I20260927 02:33:43.895854 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fadf4cc5fa46c9958d18ddf54ece4e" candidate_uuid: "e31b21f7fd594078ae3066bbb49a09c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0d8dd12754e439ca34a41122f43de11"
I20260927 02:33:43.895955 2275 raft_consensus.cc:3037] T 98fadf4cc5fa46c9958d18ddf54ece4e P f0d8dd12754e439ca34a41122f43de11 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:33:43.896219 2555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fadf4cc5fa46c9958d18ddf54ece4e" candidate_uuid: "e31b21f7fd594078ae3066bbb49a09c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74fd0b61786e4ef2abc5c35685bc05f7"
I20260927 02:33:43.896332 2555 raft_consensus.cc:3037] T 98fadf4cc5fa46c9958d18ddf54ece4e P 74fd0b61786e4ef2abc5c35685bc05f7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:33:43.897035 2275 raft_consensus.cc:2454] T 98fadf4cc5fa46c9958d18ddf54ece4e P f0d8dd12754e439ca34a41122f43de11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e31b21f7fd594078ae3066bbb49a09c1 in term 1.
I20260927 02:33:43.897332 2346 leader_election.cc:302] T 98fadf4cc5fa46c9958d18ddf54ece4e P e31b21f7fd594078ae3066bbb49a09c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e31b21f7fd594078ae3066bbb49a09c1, f0d8dd12754e439ca34a41122f43de11; no voters:
I20260927 02:33:43.897406 2555 raft_consensus.cc:2454] T 98fadf4cc5fa46c9958d18ddf54ece4e P 74fd0b61786e4ef2abc5c35685bc05f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e31b21f7fd594078ae3066bbb49a09c1 in term 1.
I20260927 02:33:43.897488 2646 raft_consensus.cc:2781] T 98fadf4cc5fa46c9958d18ddf54ece4e P e31b21f7fd594078ae3066bbb49a09c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:33:43.897552 2646 raft_consensus.cc:686] T 98fadf4cc5fa46c9958d18ddf54ece4e P e31b21f7fd594078ae3066bbb49a09c1 [term 1 LEADER]: Becoming Leader. State: Replica: e31b21f7fd594078ae3066bbb49a09c1, State: Running, Role: LEADER
I20260927 02:33:43.897688 2646 consensus_queue.cc:237] T 98fadf4cc5fa46c9958d18ddf54ece4e P e31b21f7fd594078ae3066bbb49a09c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } } peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } }
I20260927 02:33:43.899448 2118 catalog_manager.cc:5495] T 98fadf4cc5fa46c9958d18ddf54ece4e P e31b21f7fd594078ae3066bbb49a09c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to e31b21f7fd594078ae3066bbb49a09c1 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "e31b21f7fd594078ae3066bbb49a09c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } health_report { overall_health: UNKNOWN } } }
I20260927 02:33:43.901598 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:43.895347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54201 (local address 127.0.98.3:34811)
0927 02:33:43.895501 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:43.895505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:43.896655 (+ 1150us) server_negotiation.cc:407] Connection header received
0927 02:33:43.896811 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:43.896814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:43.896854 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:43.896895 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:33:43.897306 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.898095 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.900285 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.900890 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.901019 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:43.901091 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:43.901412 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 02:33:43.901479 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 02:33:43.902156 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:43.894155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34811 (local address 127.0.98.1:54201)
0927 02:33:43.896580 (+ 2425us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:43.896593 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:43.896712 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:43.897034 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:43.897036 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:43.897051 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:33:43.897196 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:43.897202 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.898262 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.898265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:43.900153 (+ 1888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:43.900164 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.901186 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.901189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:43.901289 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:43.901311 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 02:33:43.901769 (+ 458us) client_negotiation.cc:240] Negotiation successful
0927 02:33:43.902003 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2362,"mutex_wait_us":145,"thread_start_us":61,"threads_started":1}
I20260927 02:33:43.902163 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:43.895455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56837 (local address 127.0.98.2:45769)
0927 02:33:43.895627 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:43.895630 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:43.895641 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:33:43.895678 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:43.895681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:43.895722 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:43.895786 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:33:43.897627 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.898283 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.900493 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.901040 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.901128 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:43.901184 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:43.901989 (+ 805us) server_negotiation.cc:299] Negotiation successful
0927 02:33:43.902060 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260927 02:33:43.902454 2676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:43.893621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45769 (local address 127.0.98.1:56837)
0927 02:33:43.894684 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:43.894700 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:43.894791 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:43.896158 (+ 1367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:43.896162 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:43.896184 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:33:43.896325 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:43.896332 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.899118 (+ 2786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.899121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:43.900357 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:43.900367 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.901712 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.901716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:43.901857 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:43.901876 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 02:33:43.902265 (+ 389us) client_negotiation.cc:240] Negotiation successful
0927 02:33:43.902344 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":975,"mutex_wait_us":1,"thread_start_us":77,"threads_started":1}
I20260927 02:33:43.902798 2555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fadf4cc5fa46c9958d18ddf54ece4e" candidate_uuid: "f0d8dd12754e439ca34a41122f43de11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" is_pre_election: true
I20260927 02:33:43.902892 2415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fadf4cc5fa46c9958d18ddf54ece4e" candidate_uuid: "f0d8dd12754e439ca34a41122f43de11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e31b21f7fd594078ae3066bbb49a09c1" is_pre_election: true
I20260927 02:33:43.902973 2555 raft_consensus.cc:2379] T 98fadf4cc5fa46c9958d18ddf54ece4e P 74fd0b61786e4ef2abc5c35685bc05f7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0d8dd12754e439ca34a41122f43de11 in current term 1: Already voted for candidate e31b21f7fd594078ae3066bbb49a09c1 in this term.
I20260927 02:33:43.903383 2207 leader_election.cc:302] T 98fadf4cc5fa46c9958d18ddf54ece4e P f0d8dd12754e439ca34a41122f43de11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0d8dd12754e439ca34a41122f43de11; no voters: 74fd0b61786e4ef2abc5c35685bc05f7, e31b21f7fd594078ae3066bbb49a09c1
I20260927 02:33:43.903631 2630 raft_consensus.cc:2726] T 98fadf4cc5fa46c9958d18ddf54ece4e P f0d8dd12754e439ca34a41122f43de11 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 02:33:43.925796 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:43.922383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53572 (local address 127.0.98.2:45769)
0927 02:33:43.922481 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:43.922484 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:43.922574 (+ 90us) server_negotiation.cc:407] Connection header received
0927 02:33:43.922686 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:43.922688 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:43.922724 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:43.922782 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:33:43.922994 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.923473 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.924593 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.924884 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.925044 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:43.925294 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:33:43.925441 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:33:43.925505 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:43.925611 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 02:33:43.925655 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 02:33:43.933542 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:43.930850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45769 (local address 127.0.98.2:33303)
0927 02:33:43.930962 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:43.930976 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:43.931047 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:43.931231 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:43.931234 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:43.931239 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:33:43.931337 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:43.931342 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.931958 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.931961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:43.932527 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:43.932534 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.932897 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.932900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:43.933003 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:43.933026 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:33:43.933291 (+ 265us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:33:43.933296 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:33:43.933345 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 02:33:43.933408 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":16}
I20260927 02:33:43.933569 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:43.930925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33303 (local address 127.0.98.2:45769)
0927 02:33:43.931015 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:43.931019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:43.931030 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:33:43.931130 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:43.931133 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:43.931167 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:43.931221 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:33:43.931413 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.931873 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.932622 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:43.932816 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:43.932893 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:43.933139 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:33:43.933215 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:33:43.933285 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:43.933369 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 02:33:43.933419 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":38}
I20260927 02:33:43.939112 2275 raft_consensus.cc:1261] T 98fadf4cc5fa46c9958d18ddf54ece4e P f0d8dd12754e439ca34a41122f43de11 [term 1 FOLLOWER]: Refusing update from remote peer e31b21f7fd594078ae3066bbb49a09c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:33:43.939109 2555 raft_consensus.cc:1261] T 98fadf4cc5fa46c9958d18ddf54ece4e P 74fd0b61786e4ef2abc5c35685bc05f7 [term 1 FOLLOWER]: Refusing update from remote peer e31b21f7fd594078ae3066bbb49a09c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:33:43.939469 2646 consensus_queue.cc:1035] T 98fadf4cc5fa46c9958d18ddf54ece4e P e31b21f7fd594078ae3066bbb49a09c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:33:43.939604 2631 consensus_queue.cc:1035] T 98fadf4cc5fa46c9958d18ddf54ece4e P e31b21f7fd594078ae3066bbb49a09c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 02:33:43.954032 2651 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 02:33:45.506955 2118 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:33:45.507099 2118 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:33:45.509321 2118 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 307e7dee3cca4210b78b3e0eeba083a4: Sending DeleteTablet for 3 replicas of tablet 98fadf4cc5fa46c9958d18ddf54ece4e
I20260927 02:33:45.509835 2395 tablet_service.cc:1514] Processing DeleteTablet for tablet 98fadf4cc5fa46c9958d18ddf54ece4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:33:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53548
I20260927 02:33:45.509824 2255 tablet_service.cc:1514] Processing DeleteTablet for tablet 98fadf4cc5fa46c9958d18ddf54ece4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:33:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43464
I20260927 02:33:45.509856 2535 tablet_service.cc:1514] Processing DeleteTablet for tablet 98fadf4cc5fa46c9958d18ddf54ece4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:33:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44980
I20260927 02:33:45.510141 2704 tablet_replica.cc:330] T 98fadf4cc5fa46c9958d18ddf54ece4e P 74fd0b61786e4ef2abc5c35685bc05f7: stopping tablet replica
I20260927 02:33:45.510234 2702 tablet_replica.cc:330] T 98fadf4cc5fa46c9958d18ddf54ece4e P e31b21f7fd594078ae3066bbb49a09c1: stopping tablet replica
I20260927 02:33:45.510377 2702 raft_consensus.cc:2229] T 98fadf4cc5fa46c9958d18ddf54ece4e P e31b21f7fd594078ae3066bbb49a09c1 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:33:45.510265 2704 raft_consensus.cc:2229] T 98fadf4cc5fa46c9958d18ddf54ece4e P 74fd0b61786e4ef2abc5c35685bc05f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:33:45.510654 2702 raft_consensus.cc:2258] T 98fadf4cc5fa46c9958d18ddf54ece4e P e31b21f7fd594078ae3066bbb49a09c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:33:45.510756 2704 raft_consensus.cc:2258] T 98fadf4cc5fa46c9958d18ddf54ece4e P 74fd0b61786e4ef2abc5c35685bc05f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:33:45.511349 2704 ts_tablet_manager.cc:1905] T 98fadf4cc5fa46c9958d18ddf54ece4e P 74fd0b61786e4ef2abc5c35685bc05f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:33:45.511391 2702 ts_tablet_manager.cc:1905] T 98fadf4cc5fa46c9958d18ddf54ece4e P e31b21f7fd594078ae3066bbb49a09c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:33:45.512995 2703 tablet_replica.cc:330] T 98fadf4cc5fa46c9958d18ddf54ece4e P f0d8dd12754e439ca34a41122f43de11: stopping tablet replica
I20260927 02:33:45.513111 2703 raft_consensus.cc:2229] T 98fadf4cc5fa46c9958d18ddf54ece4e P f0d8dd12754e439ca34a41122f43de11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:33:45.513221 2703 raft_consensus.cc:2258] T 98fadf4cc5fa46c9958d18ddf54ece4e P f0d8dd12754e439ca34a41122f43de11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:33:45.513784 2703 ts_tablet_manager.cc:1905] T 98fadf4cc5fa46c9958d18ddf54ece4e P f0d8dd12754e439ca34a41122f43de11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:33:45.514176 2704 ts_tablet_manager.cc:1918] T 98fadf4cc5fa46c9958d18ddf54ece4e P 74fd0b61786e4ef2abc5c35685bc05f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 02:33:45.514245 2704 log.cc:1196] T 98fadf4cc5fa46c9958d18ddf54ece4e P 74fd0b61786e4ef2abc5c35685bc05f7: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-2/wal/wals/98fadf4cc5fa46c9958d18ddf54ece4e
I20260927 02:33:45.514258 2702 ts_tablet_manager.cc:1918] T 98fadf4cc5fa46c9958d18ddf54ece4e P e31b21f7fd594078ae3066bbb49a09c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 02:33:45.514334 2702 log.cc:1196] T 98fadf4cc5fa46c9958d18ddf54ece4e P e31b21f7fd594078ae3066bbb49a09c1: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-1/wal/wals/98fadf4cc5fa46c9958d18ddf54ece4e
I20260927 02:33:45.514493 2704 ts_tablet_manager.cc:1939] T 98fadf4cc5fa46c9958d18ddf54ece4e P 74fd0b61786e4ef2abc5c35685bc05f7: Deleting consensus metadata
I20260927 02:33:45.514657 2702 ts_tablet_manager.cc:1939] T 98fadf4cc5fa46c9958d18ddf54ece4e P e31b21f7fd594078ae3066bbb49a09c1: Deleting consensus metadata
I20260927 02:33:45.515257 2105 catalog_manager.cc:4841] TS e31b21f7fd594078ae3066bbb49a09c1 (127.0.98.2:45769): tablet 98fadf4cc5fa46c9958d18ddf54ece4e (table test-table [id=1930df9498b74587b8c72e6928a12c06]) successfully deleted
I20260927 02:33:45.515257 2103 catalog_manager.cc:4841] TS 74fd0b61786e4ef2abc5c35685bc05f7 (127.0.98.3:34811): tablet 98fadf4cc5fa46c9958d18ddf54ece4e (table test-table [id=1930df9498b74587b8c72e6928a12c06]) successfully deleted
I20260927 02:33:45.516033 2703 ts_tablet_manager.cc:1918] T 98fadf4cc5fa46c9958d18ddf54ece4e P f0d8dd12754e439ca34a41122f43de11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 02:33:45.516105 2703 log.cc:1196] T 98fadf4cc5fa46c9958d18ddf54ece4e P f0d8dd12754e439ca34a41122f43de11: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-0/wal/wals/98fadf4cc5fa46c9958d18ddf54ece4e
I20260927 02:33:45.516319 2703 ts_tablet_manager.cc:1939] T 98fadf4cc5fa46c9958d18ddf54ece4e P f0d8dd12754e439ca34a41122f43de11: Deleting consensus metadata
I20260927 02:33:45.516911 2102 catalog_manager.cc:4841] TS f0d8dd12754e439ca34a41122f43de11 (127.0.98.1:40513): tablet 98fadf4cc5fa46c9958d18ddf54ece4e (table test-table [id=1930df9498b74587b8c72e6928a12c06]) successfully deleted
Sep 27 02:33:45 dist-test-slave-grlk krb5kdc[2073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476425, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 02:33:45 dist-test-slave-grlk krb5kdc[2073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476425, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:33:45.536305 2712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:45.528386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33332 (local address 127.0.98.62:38251)
0927 02:33:45.528640 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:45.528650 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:45.528686 (+ 36us) server_negotiation.cc:407] Connection header received
0927 02:33:45.528839 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:45.528841 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:45.528882 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:45.529021 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:45.529656 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:45.530192 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:45.531010 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:45.531228 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:45.531310 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:45.533553 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:45.533557 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:45.533558 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:45.533579 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:45.534826 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:45.535174 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:45.535178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:45.535180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:45.535252 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:45.535483 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:45.535486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:45.535488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:45.535677 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:45.535791 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:45.536025 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 02:33:45.536142 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":84,"threads_started":1}
I20260927 02:33:45.538443 2118 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33332:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:33:45.538795 2118 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:33:45.542930 2395 tablet_service.cc:1467] Processing CreateTablet for tablet 34be4f3b5f2c4e6d81593e3f92ed49c3 (DEFAULT_TABLE table=test-table [id=2f41ade245f24476910d23d02e17dc89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:33:45.542904 2255 tablet_service.cc:1467] Processing CreateTablet for tablet 34be4f3b5f2c4e6d81593e3f92ed49c3 (DEFAULT_TABLE table=test-table [id=2f41ade245f24476910d23d02e17dc89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:33:45.543125 2535 tablet_service.cc:1467] Processing CreateTablet for tablet 34be4f3b5f2c4e6d81593e3f92ed49c3 (DEFAULT_TABLE table=test-table [id=2f41ade245f24476910d23d02e17dc89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:33:45.543212 2395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34be4f3b5f2c4e6d81593e3f92ed49c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:33:45.543215 2255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34be4f3b5f2c4e6d81593e3f92ed49c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:33:45.543342 2535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34be4f3b5f2c4e6d81593e3f92ed49c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:33:45.545732 2715 tablet_bootstrap.cc:492] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P 74fd0b61786e4ef2abc5c35685bc05f7: Bootstrap starting.
I20260927 02:33:45.545769 2713 tablet_bootstrap.cc:492] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P f0d8dd12754e439ca34a41122f43de11: Bootstrap starting.
I20260927 02:33:45.546571 2715 tablet_bootstrap.cc:654] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P 74fd0b61786e4ef2abc5c35685bc05f7: Neither blocks nor log segments found. Creating new log.
I20260927 02:33:45.546583 2713 tablet_bootstrap.cc:654] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P f0d8dd12754e439ca34a41122f43de11: Neither blocks nor log segments found. Creating new log.
I20260927 02:33:45.547147 2714 tablet_bootstrap.cc:492] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P e31b21f7fd594078ae3066bbb49a09c1: Bootstrap starting.
I20260927 02:33:45.547636 2713 tablet_bootstrap.cc:492] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P f0d8dd12754e439ca34a41122f43de11: No bootstrap required, opened a new log
I20260927 02:33:45.547711 2713 ts_tablet_manager.cc:1397] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P f0d8dd12754e439ca34a41122f43de11: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:33:45.547896 2714 tablet_bootstrap.cc:654] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P e31b21f7fd594078ae3066bbb49a09c1: Neither blocks nor log segments found. Creating new log.
I20260927 02:33:45.548103 2713 raft_consensus.cc:348] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P f0d8dd12754e439ca34a41122f43de11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } } peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } }
I20260927 02:33:45.548193 2713 raft_consensus.cc:374] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P f0d8dd12754e439ca34a41122f43de11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:33:45.548223 2713 raft_consensus.cc:729] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P f0d8dd12754e439ca34a41122f43de11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0d8dd12754e439ca34a41122f43de11, State: Initialized, Role: FOLLOWER
I20260927 02:33:45.548358 2713 consensus_queue.cc:260] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P f0d8dd12754e439ca34a41122f43de11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } } peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } }
I20260927 02:33:45.548736 2713 ts_tablet_manager.cc:1428] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P f0d8dd12754e439ca34a41122f43de11: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:33:45.548785 2715 tablet_bootstrap.cc:492] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P 74fd0b61786e4ef2abc5c35685bc05f7: No bootstrap required, opened a new log
I20260927 02:33:45.548867 2715 ts_tablet_manager.cc:1397] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P 74fd0b61786e4ef2abc5c35685bc05f7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 02:33:45.549081 2714 tablet_bootstrap.cc:492] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P e31b21f7fd594078ae3066bbb49a09c1: No bootstrap required, opened a new log
I20260927 02:33:45.549139 2714 ts_tablet_manager.cc:1397] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P e31b21f7fd594078ae3066bbb49a09c1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:33:45.549285 2715 raft_consensus.cc:348] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P 74fd0b61786e4ef2abc5c35685bc05f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } } peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } }
I20260927 02:33:45.549381 2715 raft_consensus.cc:374] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P 74fd0b61786e4ef2abc5c35685bc05f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:33:45.549415 2715 raft_consensus.cc:729] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P 74fd0b61786e4ef2abc5c35685bc05f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74fd0b61786e4ef2abc5c35685bc05f7, State: Initialized, Role: FOLLOWER
I20260927 02:33:45.549477 2714 raft_consensus.cc:348] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P e31b21f7fd594078ae3066bbb49a09c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } } peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } }
I20260927 02:33:45.549541 2715 consensus_queue.cc:260] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P 74fd0b61786e4ef2abc5c35685bc05f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } } peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } }
I20260927 02:33:45.549587 2714 raft_consensus.cc:374] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P e31b21f7fd594078ae3066bbb49a09c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:33:45.549621 2714 raft_consensus.cc:729] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P e31b21f7fd594078ae3066bbb49a09c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e31b21f7fd594078ae3066bbb49a09c1, State: Initialized, Role: FOLLOWER
I20260927 02:33:45.549728 2714 consensus_queue.cc:260] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P e31b21f7fd594078ae3066bbb49a09c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } } peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } }
I20260927 02:33:45.549865 2715 ts_tablet_manager.cc:1428] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P 74fd0b61786e4ef2abc5c35685bc05f7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:33:45.549979 2714 ts_tablet_manager.cc:1428] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P e31b21f7fd594078ae3066bbb49a09c1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 02:33:45.707661 2697 raft_consensus.cc:482] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P e31b21f7fd594078ae3066bbb49a09c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:33:45.707813 2697 raft_consensus.cc:504] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P e31b21f7fd594078ae3066bbb49a09c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } } peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } }
I20260927 02:33:45.708235 2697 leader_election.cc:288] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P e31b21f7fd594078ae3066bbb49a09c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0d8dd12754e439ca34a41122f43de11 (127.0.98.1:40513), 74fd0b61786e4ef2abc5c35685bc05f7 (127.0.98.3:34811)
I20260927 02:33:45.708819 2554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34be4f3b5f2c4e6d81593e3f92ed49c3" candidate_uuid: "e31b21f7fd594078ae3066bbb49a09c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" is_pre_election: true
I20260927 02:33:45.708817 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34be4f3b5f2c4e6d81593e3f92ed49c3" candidate_uuid: "e31b21f7fd594078ae3066bbb49a09c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0d8dd12754e439ca34a41122f43de11" is_pre_election: true
I20260927 02:33:45.709024 2554 raft_consensus.cc:2454] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P 74fd0b61786e4ef2abc5c35685bc05f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e31b21f7fd594078ae3066bbb49a09c1 in term 0.
I20260927 02:33:45.709016 2275 raft_consensus.cc:2454] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P f0d8dd12754e439ca34a41122f43de11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e31b21f7fd594078ae3066bbb49a09c1 in term 0.
I20260927 02:33:45.709357 2346 leader_election.cc:302] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P e31b21f7fd594078ae3066bbb49a09c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e31b21f7fd594078ae3066bbb49a09c1, f0d8dd12754e439ca34a41122f43de11; no voters:
I20260927 02:33:45.709534 2697 raft_consensus.cc:2781] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P e31b21f7fd594078ae3066bbb49a09c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:33:45.709595 2697 raft_consensus.cc:482] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P e31b21f7fd594078ae3066bbb49a09c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:33:45.709645 2697 raft_consensus.cc:3037] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P e31b21f7fd594078ae3066bbb49a09c1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:33:45.710619 2697 raft_consensus.cc:504] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P e31b21f7fd594078ae3066bbb49a09c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } } peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } }
I20260927 02:33:45.710938 2697 leader_election.cc:288] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P e31b21f7fd594078ae3066bbb49a09c1 [CANDIDATE]: Term 1 election: Requested vote from peers f0d8dd12754e439ca34a41122f43de11 (127.0.98.1:40513), 74fd0b61786e4ef2abc5c35685bc05f7 (127.0.98.3:34811)
I20260927 02:33:45.711315 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34be4f3b5f2c4e6d81593e3f92ed49c3" candidate_uuid: "e31b21f7fd594078ae3066bbb49a09c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0d8dd12754e439ca34a41122f43de11"
I20260927 02:33:45.711315 2554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34be4f3b5f2c4e6d81593e3f92ed49c3" candidate_uuid: "e31b21f7fd594078ae3066bbb49a09c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74fd0b61786e4ef2abc5c35685bc05f7"
I20260927 02:33:45.711440 2275 raft_consensus.cc:3037] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P f0d8dd12754e439ca34a41122f43de11 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:33:45.711440 2554 raft_consensus.cc:3037] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P 74fd0b61786e4ef2abc5c35685bc05f7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:33:45.712482 2275 raft_consensus.cc:2454] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P f0d8dd12754e439ca34a41122f43de11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e31b21f7fd594078ae3066bbb49a09c1 in term 1.
I20260927 02:33:45.712491 2554 raft_consensus.cc:2454] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P 74fd0b61786e4ef2abc5c35685bc05f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e31b21f7fd594078ae3066bbb49a09c1 in term 1.
I20260927 02:33:45.712741 2346 leader_election.cc:302] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P e31b21f7fd594078ae3066bbb49a09c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e31b21f7fd594078ae3066bbb49a09c1, f0d8dd12754e439ca34a41122f43de11; no voters:
I20260927 02:33:45.712899 2697 raft_consensus.cc:2781] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P e31b21f7fd594078ae3066bbb49a09c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:33:45.713014 2697 raft_consensus.cc:686] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P e31b21f7fd594078ae3066bbb49a09c1 [term 1 LEADER]: Becoming Leader. State: Replica: e31b21f7fd594078ae3066bbb49a09c1, State: Running, Role: LEADER
I20260927 02:33:45.713160 2697 consensus_queue.cc:237] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P e31b21f7fd594078ae3066bbb49a09c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } } peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } }
I20260927 02:33:45.714793 2117 catalog_manager.cc:5495] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P e31b21f7fd594078ae3066bbb49a09c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to e31b21f7fd594078ae3066bbb49a09c1 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "e31b21f7fd594078ae3066bbb49a09c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } health_report { overall_health: UNKNOWN } } }
I20260927 02:33:45.738886 2723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:45.735920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53576 (local address 127.0.98.2:45769)
0927 02:33:45.736152 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:45.736160 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:45.736169 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:33:45.736208 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:45.736211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:45.736257 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:45.736329 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:33:45.736570 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:45.737148 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:45.737944 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:45.738182 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:45.738261 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:45.738491 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:33:45.738563 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:33:45.738618 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:45.738704 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 02:33:45.738750 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":89,"threads_started":1}
I20260927 02:33:45.743590 2554 raft_consensus.cc:1261] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P 74fd0b61786e4ef2abc5c35685bc05f7 [term 1 FOLLOWER]: Refusing update from remote peer e31b21f7fd594078ae3066bbb49a09c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:33:45.743595 2275 raft_consensus.cc:1261] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P f0d8dd12754e439ca34a41122f43de11 [term 1 FOLLOWER]: Refusing update from remote peer e31b21f7fd594078ae3066bbb49a09c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:33:45.743889 2685 consensus_queue.cc:1035] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P e31b21f7fd594078ae3066bbb49a09c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:33:45.744046 2682 consensus_queue.cc:1035] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P e31b21f7fd594078ae3066bbb49a09c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:33:45.770335 2116 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:33:45.770489 2116 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:33:45.772115 2116 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 307e7dee3cca4210b78b3e0eeba083a4: Sending DeleteTablet for 3 replicas of tablet 34be4f3b5f2c4e6d81593e3f92ed49c3
I20260927 02:33:45.772663 2395 tablet_service.cc:1514] Processing DeleteTablet for tablet 34be4f3b5f2c4e6d81593e3f92ed49c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:33:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53548
I20260927 02:33:45.772670 2255 tablet_service.cc:1514] Processing DeleteTablet for tablet 34be4f3b5f2c4e6d81593e3f92ed49c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:33:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43464
I20260927 02:33:45.772771 2535 tablet_service.cc:1514] Processing DeleteTablet for tablet 34be4f3b5f2c4e6d81593e3f92ed49c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:33:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44980
I20260927 02:33:45.772866 2702 tablet_replica.cc:330] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P e31b21f7fd594078ae3066bbb49a09c1: stopping tablet replica
I20260927 02:33:45.772962 2703 tablet_replica.cc:330] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P f0d8dd12754e439ca34a41122f43de11: stopping tablet replica
I20260927 02:33:45.773010 2702 raft_consensus.cc:2229] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P e31b21f7fd594078ae3066bbb49a09c1 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:33:45.773099 2703 raft_consensus.cc:2229] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P f0d8dd12754e439ca34a41122f43de11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:33:45.773180 2702 raft_consensus.cc:2258] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P e31b21f7fd594078ae3066bbb49a09c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:33:45.773204 2703 raft_consensus.cc:2258] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P f0d8dd12754e439ca34a41122f43de11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:33:45.773355 2704 tablet_replica.cc:330] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P 74fd0b61786e4ef2abc5c35685bc05f7: stopping tablet replica
I20260927 02:33:45.773466 2704 raft_consensus.cc:2229] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P 74fd0b61786e4ef2abc5c35685bc05f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:33:45.773567 2704 raft_consensus.cc:2258] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P 74fd0b61786e4ef2abc5c35685bc05f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:33:45.774138 2702 ts_tablet_manager.cc:1905] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P e31b21f7fd594078ae3066bbb49a09c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:33:45.774253 2704 ts_tablet_manager.cc:1905] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P 74fd0b61786e4ef2abc5c35685bc05f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:33:45.776247 2702 ts_tablet_manager.cc:1918] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P e31b21f7fd594078ae3066bbb49a09c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:33:45.776310 2702 log.cc:1196] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P e31b21f7fd594078ae3066bbb49a09c1: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-1/wal/wals/34be4f3b5f2c4e6d81593e3f92ed49c3
I20260927 02:33:45.776409 2704 ts_tablet_manager.cc:1918] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P 74fd0b61786e4ef2abc5c35685bc05f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:33:45.776485 2704 log.cc:1196] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P 74fd0b61786e4ef2abc5c35685bc05f7: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-2/wal/wals/34be4f3b5f2c4e6d81593e3f92ed49c3
I20260927 02:33:45.776508 2702 ts_tablet_manager.cc:1939] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P e31b21f7fd594078ae3066bbb49a09c1: Deleting consensus metadata
I20260927 02:33:45.776691 2704 ts_tablet_manager.cc:1939] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P 74fd0b61786e4ef2abc5c35685bc05f7: Deleting consensus metadata
I20260927 02:33:45.777123 2105 catalog_manager.cc:4841] TS e31b21f7fd594078ae3066bbb49a09c1 (127.0.98.2:45769): tablet 34be4f3b5f2c4e6d81593e3f92ed49c3 (table test-table [id=2f41ade245f24476910d23d02e17dc89]) successfully deleted
I20260927 02:33:45.777293 2103 catalog_manager.cc:4841] TS 74fd0b61786e4ef2abc5c35685bc05f7 (127.0.98.3:34811): tablet 34be4f3b5f2c4e6d81593e3f92ed49c3 (table test-table [id=2f41ade245f24476910d23d02e17dc89]) successfully deleted
I20260927 02:33:45.777545 2703 ts_tablet_manager.cc:1905] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P f0d8dd12754e439ca34a41122f43de11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:33:45.779347 2703 ts_tablet_manager.cc:1918] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P f0d8dd12754e439ca34a41122f43de11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:33:45.779424 2703 log.cc:1196] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P f0d8dd12754e439ca34a41122f43de11: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-0/wal/wals/34be4f3b5f2c4e6d81593e3f92ed49c3
I20260927 02:33:45.779580 2703 ts_tablet_manager.cc:1939] T 34be4f3b5f2c4e6d81593e3f92ed49c3 P f0d8dd12754e439ca34a41122f43de11: Deleting consensus metadata
I20260927 02:33:45.779836 2116 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33332:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:33:45.780244 2116 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:33:45.780447 2102 catalog_manager.cc:4841] TS f0d8dd12754e439ca34a41122f43de11 (127.0.98.1:40513): tablet 34be4f3b5f2c4e6d81593e3f92ed49c3 (table test-table [id=2f41ade245f24476910d23d02e17dc89]) successfully deleted
I20260927 02:33:45.784521 2255 tablet_service.cc:1467] Processing CreateTablet for tablet 23aa0242437743eea0babdc341e4972f (DEFAULT_TABLE table=test-table [id=b84bb6472a024a9891d92bdc0dceaf2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:33:45.784606 2395 tablet_service.cc:1467] Processing CreateTablet for tablet 23aa0242437743eea0babdc341e4972f (DEFAULT_TABLE table=test-table [id=b84bb6472a024a9891d92bdc0dceaf2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:33:45.784788 2255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23aa0242437743eea0babdc341e4972f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:33:45.784767 2535 tablet_service.cc:1467] Processing CreateTablet for tablet 23aa0242437743eea0babdc341e4972f (DEFAULT_TABLE table=test-table [id=b84bb6472a024a9891d92bdc0dceaf2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:33:45.784860 2395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23aa0242437743eea0babdc341e4972f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:33:45.785029 2535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23aa0242437743eea0babdc341e4972f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:33:45.786507 2713 tablet_bootstrap.cc:492] T 23aa0242437743eea0babdc341e4972f P f0d8dd12754e439ca34a41122f43de11: Bootstrap starting.
I20260927 02:33:45.787075 2715 tablet_bootstrap.cc:492] T 23aa0242437743eea0babdc341e4972f P 74fd0b61786e4ef2abc5c35685bc05f7: Bootstrap starting.
I20260927 02:33:45.787313 2713 tablet_bootstrap.cc:654] T 23aa0242437743eea0babdc341e4972f P f0d8dd12754e439ca34a41122f43de11: Neither blocks nor log segments found. Creating new log.
I20260927 02:33:45.787810 2715 tablet_bootstrap.cc:654] T 23aa0242437743eea0babdc341e4972f P 74fd0b61786e4ef2abc5c35685bc05f7: Neither blocks nor log segments found. Creating new log.
I20260927 02:33:45.788295 2714 tablet_bootstrap.cc:492] T 23aa0242437743eea0babdc341e4972f P e31b21f7fd594078ae3066bbb49a09c1: Bootstrap starting.
I20260927 02:33:45.788419 2713 tablet_bootstrap.cc:492] T 23aa0242437743eea0babdc341e4972f P f0d8dd12754e439ca34a41122f43de11: No bootstrap required, opened a new log
I20260927 02:33:45.788503 2713 ts_tablet_manager.cc:1397] T 23aa0242437743eea0babdc341e4972f P f0d8dd12754e439ca34a41122f43de11: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:33:45.788739 2715 tablet_bootstrap.cc:492] T 23aa0242437743eea0babdc341e4972f P 74fd0b61786e4ef2abc5c35685bc05f7: No bootstrap required, opened a new log
I20260927 02:33:45.788823 2715 ts_tablet_manager.cc:1397] T 23aa0242437743eea0babdc341e4972f P 74fd0b61786e4ef2abc5c35685bc05f7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:33:45.788975 2713 raft_consensus.cc:348] T 23aa0242437743eea0babdc341e4972f P f0d8dd12754e439ca34a41122f43de11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } } peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } }
I20260927 02:33:45.789088 2713 raft_consensus.cc:374] T 23aa0242437743eea0babdc341e4972f P f0d8dd12754e439ca34a41122f43de11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:33:45.789119 2713 raft_consensus.cc:729] T 23aa0242437743eea0babdc341e4972f P f0d8dd12754e439ca34a41122f43de11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0d8dd12754e439ca34a41122f43de11, State: Initialized, Role: FOLLOWER
I20260927 02:33:45.789234 2713 consensus_queue.cc:260] T 23aa0242437743eea0babdc341e4972f P f0d8dd12754e439ca34a41122f43de11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } } peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } }
I20260927 02:33:45.789227 2715 raft_consensus.cc:348] T 23aa0242437743eea0babdc341e4972f P 74fd0b61786e4ef2abc5c35685bc05f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } } peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } }
I20260927 02:33:45.789400 2715 raft_consensus.cc:374] T 23aa0242437743eea0babdc341e4972f P 74fd0b61786e4ef2abc5c35685bc05f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:33:45.789469 2715 raft_consensus.cc:729] T 23aa0242437743eea0babdc341e4972f P 74fd0b61786e4ef2abc5c35685bc05f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74fd0b61786e4ef2abc5c35685bc05f7, State: Initialized, Role: FOLLOWER
I20260927 02:33:45.789600 2713 ts_tablet_manager.cc:1428] T 23aa0242437743eea0babdc341e4972f P f0d8dd12754e439ca34a41122f43de11: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:33:45.789669 2714 tablet_bootstrap.cc:654] T 23aa0242437743eea0babdc341e4972f P e31b21f7fd594078ae3066bbb49a09c1: Neither blocks nor log segments found. Creating new log.
I20260927 02:33:45.789621 2715 consensus_queue.cc:260] T 23aa0242437743eea0babdc341e4972f P 74fd0b61786e4ef2abc5c35685bc05f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } } peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } }
I20260927 02:33:45.790218 2715 ts_tablet_manager.cc:1428] T 23aa0242437743eea0babdc341e4972f P 74fd0b61786e4ef2abc5c35685bc05f7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:33:45.790910 2714 tablet_bootstrap.cc:492] T 23aa0242437743eea0babdc341e4972f P e31b21f7fd594078ae3066bbb49a09c1: No bootstrap required, opened a new log
I20260927 02:33:45.790988 2714 ts_tablet_manager.cc:1397] T 23aa0242437743eea0babdc341e4972f P e31b21f7fd594078ae3066bbb49a09c1: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 02:33:45.791538 2714 raft_consensus.cc:348] T 23aa0242437743eea0babdc341e4972f P e31b21f7fd594078ae3066bbb49a09c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } } peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } }
I20260927 02:33:45.791848 2714 raft_consensus.cc:374] T 23aa0242437743eea0babdc341e4972f P e31b21f7fd594078ae3066bbb49a09c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:33:45.791894 2714 raft_consensus.cc:729] T 23aa0242437743eea0babdc341e4972f P e31b21f7fd594078ae3066bbb49a09c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e31b21f7fd594078ae3066bbb49a09c1, State: Initialized, Role: FOLLOWER
I20260927 02:33:45.792024 2714 consensus_queue.cc:260] T 23aa0242437743eea0babdc341e4972f P e31b21f7fd594078ae3066bbb49a09c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } } peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } }
I20260927 02:33:45.792959 2714 ts_tablet_manager.cc:1428] T 23aa0242437743eea0babdc341e4972f P e31b21f7fd594078ae3066bbb49a09c1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:33:45.863365 2720 raft_consensus.cc:482] T 23aa0242437743eea0babdc341e4972f P 74fd0b61786e4ef2abc5c35685bc05f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:33:45.863524 2720 raft_consensus.cc:504] T 23aa0242437743eea0babdc341e4972f P 74fd0b61786e4ef2abc5c35685bc05f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } } peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } }
I20260927 02:33:45.863895 2720 leader_election.cc:288] T 23aa0242437743eea0babdc341e4972f P 74fd0b61786e4ef2abc5c35685bc05f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0d8dd12754e439ca34a41122f43de11 (127.0.98.1:40513), e31b21f7fd594078ae3066bbb49a09c1 (127.0.98.2:45769)
I20260927 02:33:45.864388 2415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23aa0242437743eea0babdc341e4972f" candidate_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e31b21f7fd594078ae3066bbb49a09c1" is_pre_election: true
I20260927 02:33:45.864460 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23aa0242437743eea0babdc341e4972f" candidate_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0d8dd12754e439ca34a41122f43de11" is_pre_election: true
I20260927 02:33:45.864543 2415 raft_consensus.cc:2454] T 23aa0242437743eea0babdc341e4972f P e31b21f7fd594078ae3066bbb49a09c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74fd0b61786e4ef2abc5c35685bc05f7 in term 0.
I20260927 02:33:45.864653 2275 raft_consensus.cc:2454] T 23aa0242437743eea0babdc341e4972f P f0d8dd12754e439ca34a41122f43de11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74fd0b61786e4ef2abc5c35685bc05f7 in term 0.
I20260927 02:33:45.864897 2489 leader_election.cc:302] T 23aa0242437743eea0babdc341e4972f P 74fd0b61786e4ef2abc5c35685bc05f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74fd0b61786e4ef2abc5c35685bc05f7, e31b21f7fd594078ae3066bbb49a09c1; no voters:
I20260927 02:33:45.865155 2720 raft_consensus.cc:2781] T 23aa0242437743eea0babdc341e4972f P 74fd0b61786e4ef2abc5c35685bc05f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:33:45.865224 2720 raft_consensus.cc:482] T 23aa0242437743eea0babdc341e4972f P 74fd0b61786e4ef2abc5c35685bc05f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:33:45.865271 2720 raft_consensus.cc:3037] T 23aa0242437743eea0babdc341e4972f P 74fd0b61786e4ef2abc5c35685bc05f7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:33:45.866204 2720 raft_consensus.cc:504] T 23aa0242437743eea0babdc341e4972f P 74fd0b61786e4ef2abc5c35685bc05f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } } peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } }
I20260927 02:33:45.866488 2720 leader_election.cc:288] T 23aa0242437743eea0babdc341e4972f P 74fd0b61786e4ef2abc5c35685bc05f7 [CANDIDATE]: Term 1 election: Requested vote from peers f0d8dd12754e439ca34a41122f43de11 (127.0.98.1:40513), e31b21f7fd594078ae3066bbb49a09c1 (127.0.98.2:45769)
I20260927 02:33:45.866804 2415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23aa0242437743eea0babdc341e4972f" candidate_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e31b21f7fd594078ae3066bbb49a09c1"
I20260927 02:33:45.866809 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23aa0242437743eea0babdc341e4972f" candidate_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0d8dd12754e439ca34a41122f43de11"
I20260927 02:33:45.866907 2415 raft_consensus.cc:3037] T 23aa0242437743eea0babdc341e4972f P e31b21f7fd594078ae3066bbb49a09c1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:33:45.866915 2275 raft_consensus.cc:3037] T 23aa0242437743eea0babdc341e4972f P f0d8dd12754e439ca34a41122f43de11 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:33:45.867728 2275 raft_consensus.cc:2454] T 23aa0242437743eea0babdc341e4972f P f0d8dd12754e439ca34a41122f43de11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74fd0b61786e4ef2abc5c35685bc05f7 in term 1.
I20260927 02:33:45.867820 2415 raft_consensus.cc:2454] T 23aa0242437743eea0babdc341e4972f P e31b21f7fd594078ae3066bbb49a09c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74fd0b61786e4ef2abc5c35685bc05f7 in term 1.
I20260927 02:33:45.867995 2486 leader_election.cc:302] T 23aa0242437743eea0babdc341e4972f P 74fd0b61786e4ef2abc5c35685bc05f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74fd0b61786e4ef2abc5c35685bc05f7, f0d8dd12754e439ca34a41122f43de11; no voters:
I20260927 02:33:45.868142 2720 raft_consensus.cc:2781] T 23aa0242437743eea0babdc341e4972f P 74fd0b61786e4ef2abc5c35685bc05f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:33:45.868315 2720 raft_consensus.cc:686] T 23aa0242437743eea0babdc341e4972f P 74fd0b61786e4ef2abc5c35685bc05f7 [term 1 LEADER]: Becoming Leader. State: Replica: 74fd0b61786e4ef2abc5c35685bc05f7, State: Running, Role: LEADER
I20260927 02:33:45.868445 2720 consensus_queue.cc:237] T 23aa0242437743eea0babdc341e4972f P 74fd0b61786e4ef2abc5c35685bc05f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } } peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } }
I20260927 02:33:45.869948 2116 catalog_manager.cc:5495] T 23aa0242437743eea0babdc341e4972f P 74fd0b61786e4ef2abc5c35685bc05f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74fd0b61786e4ef2abc5c35685bc05f7 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74fd0b61786e4ef2abc5c35685bc05f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } health_report { overall_health: HEALTHY } } }
I20260927 02:33:45.894874 2732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:45.891887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45004 (local address 127.0.98.3:34811)
0927 02:33:45.892194 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:45.892200 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:45.892210 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:33:45.892242 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:45.892244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:45.892281 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:45.892351 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:33:45.892584 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:45.893153 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:45.893935 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:45.894156 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:45.894240 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:45.894455 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:33:45.894522 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:33:45.894578 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:45.894682 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 02:33:45.894732 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":113,"threads_started":1}
I20260927 02:33:45.900189 2415 raft_consensus.cc:1261] T 23aa0242437743eea0babdc341e4972f P e31b21f7fd594078ae3066bbb49a09c1 [term 1 FOLLOWER]: Refusing update from remote peer 74fd0b61786e4ef2abc5c35685bc05f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:33:45.900223 2275 raft_consensus.cc:1261] T 23aa0242437743eea0babdc341e4972f P f0d8dd12754e439ca34a41122f43de11 [term 1 FOLLOWER]: Refusing update from remote peer 74fd0b61786e4ef2abc5c35685bc05f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:33:45.900579 2730 consensus_queue.cc:1035] T 23aa0242437743eea0babdc341e4972f P 74fd0b61786e4ef2abc5c35685bc05f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0d8dd12754e439ca34a41122f43de11" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:33:45.900727 2720 consensus_queue.cc:1035] T 23aa0242437743eea0babdc341e4972f P 74fd0b61786e4ef2abc5c35685bc05f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e31b21f7fd594078ae3066bbb49a09c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 02:33:45.913686 2686 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 02:33:47.460500 2116 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:33:47.460654 2116 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:33:47.462976 2116 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 307e7dee3cca4210b78b3e0eeba083a4: Sending DeleteTablet for 3 replicas of tablet 23aa0242437743eea0babdc341e4972f
I20260927 02:33:47.463495 2395 tablet_service.cc:1514] Processing DeleteTablet for tablet 23aa0242437743eea0babdc341e4972f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:33:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53548
I20260927 02:33:47.463527 2255 tablet_service.cc:1514] Processing DeleteTablet for tablet 23aa0242437743eea0babdc341e4972f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:33:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43464
I20260927 02:33:47.463588 2535 tablet_service.cc:1514] Processing DeleteTablet for tablet 23aa0242437743eea0babdc341e4972f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:33:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44980
I20260927 02:33:47.463972 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 2190
I20260927 02:33:47.463994 2759 tablet_replica.cc:330] T 23aa0242437743eea0babdc341e4972f P f0d8dd12754e439ca34a41122f43de11: stopping tablet replica
I20260927 02:33:47.464496 2760 tablet_replica.cc:330] T 23aa0242437743eea0babdc341e4972f P 74fd0b61786e4ef2abc5c35685bc05f7: stopping tablet replica
I20260927 02:33:47.464641 2760 raft_consensus.cc:2229] T 23aa0242437743eea0babdc341e4972f P 74fd0b61786e4ef2abc5c35685bc05f7 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:33:47.464802 2758 tablet_replica.cc:330] T 23aa0242437743eea0babdc341e4972f P e31b21f7fd594078ae3066bbb49a09c1: stopping tablet replica
I20260927 02:33:47.464819 2760 raft_consensus.cc:2258] T 23aa0242437743eea0babdc341e4972f P 74fd0b61786e4ef2abc5c35685bc05f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:33:47.465137 2758 raft_consensus.cc:2229] T 23aa0242437743eea0babdc341e4972f P e31b21f7fd594078ae3066bbb49a09c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:33:47.465262 2758 raft_consensus.cc:2258] T 23aa0242437743eea0babdc341e4972f P e31b21f7fd594078ae3066bbb49a09c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:33:47.465858 2758 ts_tablet_manager.cc:1905] T 23aa0242437743eea0babdc341e4972f P e31b21f7fd594078ae3066bbb49a09c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:33:47.468183 2758 ts_tablet_manager.cc:1918] T 23aa0242437743eea0babdc341e4972f P e31b21f7fd594078ae3066bbb49a09c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 02:33:47.468250 2758 log.cc:1196] T 23aa0242437743eea0babdc341e4972f P e31b21f7fd594078ae3066bbb49a09c1: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-1/wal/wals/23aa0242437743eea0babdc341e4972f
I20260927 02:33:47.468478 2758 ts_tablet_manager.cc:1939] T 23aa0242437743eea0babdc341e4972f P e31b21f7fd594078ae3066bbb49a09c1: Deleting consensus metadata
I20260927 02:33:47.469151 2105 catalog_manager.cc:4841] TS e31b21f7fd594078ae3066bbb49a09c1 (127.0.98.2:45769): tablet 23aa0242437743eea0babdc341e4972f (table test-table [id=b84bb6472a024a9891d92bdc0dceaf2c]) successfully deleted
I20260927 02:33:47.469390 2760 ts_tablet_manager.cc:1905] T 23aa0242437743eea0babdc341e4972f P 74fd0b61786e4ef2abc5c35685bc05f7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 02:33:47.470068 2102 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40513 (error 108)
I20260927 02:33:47.471436 2760 ts_tablet_manager.cc:1918] T 23aa0242437743eea0babdc341e4972f P 74fd0b61786e4ef2abc5c35685bc05f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 02:33:47.471494 2760 log.cc:1196] T 23aa0242437743eea0babdc341e4972f P 74fd0b61786e4ef2abc5c35685bc05f7: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476410933508-392-0/minicluster-data/ts-2/wal/wals/23aa0242437743eea0babdc341e4972f
I20260927 02:33:47.471655 2760 ts_tablet_manager.cc:1939] T 23aa0242437743eea0babdc341e4972f P 74fd0b61786e4ef2abc5c35685bc05f7: Deleting consensus metadata
I20260927 02:33:47.472249 2103 catalog_manager.cc:4841] TS 74fd0b61786e4ef2abc5c35685bc05f7 (127.0.98.3:34811): tablet 23aa0242437743eea0babdc341e4972f (table test-table [id=b84bb6472a024a9891d92bdc0dceaf2c]) successfully deleted
I20260927 02:33:47.473135 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 2330
I20260927 02:33:47.478821 2762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:47.474274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40513 (local address 127.0.0.1:43474)
0927 02:33:47.478646 (+ 4372us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:47.478725 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4296,"thread_start_us":98,"threads_started":1}
W20260927 02:33:47.478920 2102 catalog_manager.cc:4584] TS f0d8dd12754e439ca34a41122f43de11 (127.0.98.1:40513): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 23aa0242437743eea0babdc341e4972f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40513: connect: Connection refused (error 111)
I20260927 02:33:47.479096 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 2470
I20260927 02:33:47.484675 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 2086
2026-09-27T02:33:47Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5877 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:33:47 dist-test-slave-grlk krb5kdc[2765](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:33:47 dist-test-slave-grlk krb5kdc[2765](info): set up 2 sockets
Sep 27 02:33:47 dist-test-slave-grlk krb5kdc[2765](info): commencing operation
krb5kdc: starting...
W20260927 02:33:48.452602 392 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/krb5kdc/test-user.keytab.
Sep 27 02:33:48 dist-test-slave-grlk krb5kdc[2765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:33:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:33:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:33:48.557271 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37293
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37293
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:33:48.659349 2778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:33:48.659574 2778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:33:48.659611 2778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:33:48.663482 2778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:33:48.663530 2778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:33:48.663548 2778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:33:48.663563 2778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:33:48.663576 2778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:33:48.668447 2778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:33:48.669319 2778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:33:48.670176 2778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:33:48.677978 2784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:48.677971 2783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:48.677971 2786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:48.678081 2778 server_base.cc:1034] running on GCE node
I20260927 02:33:48.678821 2778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:33:48.679868 2778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:33:48.681020 2778 hybrid_clock.cc:648] HybridClock initialized: now 1790476428681011 us; error 36 us; skew 500 ppm
Sep 27 02:33:48 dist-test-slave-grlk krb5kdc[2765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476428, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:33:48.686738 2778 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:33:48.687528 2778 webserver.cc:460] Webserver started at http://127.0.98.62:43105/ using document root <none> and password file <none>
I20260927 02:33:48.688055 2778 fs_manager.cc:359] Metadata directory not provided
I20260927 02:33:48.688113 2778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:33:48.688265 2778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:33:48.690213 2778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "6135da2fa88348d1ba752d0b88a402d4"
format_stamp: "Formatted at 2026-09-27 02:33:48 on dist-test-slave-grlk"
server_key: "71f436a12a30456173545ae33f798127"
server_key_iv: "5a2b1d4bfbadd7380448288892b05257"
server_key_version: "encryptionkey@0"
I20260927 02:33:48.690694 2778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "6135da2fa88348d1ba752d0b88a402d4"
format_stamp: "Formatted at 2026-09-27 02:33:48 on dist-test-slave-grlk"
server_key: "71f436a12a30456173545ae33f798127"
server_key_iv: "5a2b1d4bfbadd7380448288892b05257"
server_key_version: "encryptionkey@0"
I20260927 02:33:48.694064 2778 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:33:48.696245 2793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:48.697441 2778 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:33:48.697540 2778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "6135da2fa88348d1ba752d0b88a402d4"
format_stamp: "Formatted at 2026-09-27 02:33:48 on dist-test-slave-grlk"
server_key: "71f436a12a30456173545ae33f798127"
server_key_iv: "5a2b1d4bfbadd7380448288892b05257"
server_key_version: "encryptionkey@0"
I20260927 02:33:48.697630 2778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:33:48.715497 2778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:33:48.718304 2778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:33:48.718483 2778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:33:48.724802 2778 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37293
I20260927 02:33:48.724808 2845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37293 every 8 connection(s)
I20260927 02:33:48.725708 2778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:33:48.728284 2846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:33:48.731765 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 2778
I20260927 02:33:48.731849 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:33:48.732100 392 external_mini_cluster.cc:1428] Setting key 5bde1c8b001a6f4b597e70c91553ab0d
I20260927 02:33:48.733565 2846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6135da2fa88348d1ba752d0b88a402d4: Bootstrap starting.
I20260927 02:33:48.736060 2846 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6135da2fa88348d1ba752d0b88a402d4: Neither blocks nor log segments found. Creating new log.
I20260927 02:33:48.736894 2846 log.cc:824] T 00000000000000000000000000000000 P 6135da2fa88348d1ba752d0b88a402d4: Log is configured to *not* fsync() on all Append() calls
I20260927 02:33:48.738710 2846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6135da2fa88348d1ba752d0b88a402d4: No bootstrap required, opened a new log
Sep 27 02:33:48 dist-test-slave-grlk krb5kdc[2765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:33:48.742575 2846 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6135da2fa88348d1ba752d0b88a402d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6135da2fa88348d1ba752d0b88a402d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37293 } }
I20260927 02:33:48.742767 2846 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6135da2fa88348d1ba752d0b88a402d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:33:48.742799 2846 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6135da2fa88348d1ba752d0b88a402d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6135da2fa88348d1ba752d0b88a402d4, State: Initialized, Role: FOLLOWER
I20260927 02:33:48.743211 2846 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6135da2fa88348d1ba752d0b88a402d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6135da2fa88348d1ba752d0b88a402d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37293 } }
I20260927 02:33:48.743363 2846 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6135da2fa88348d1ba752d0b88a402d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:33:48.743430 2846 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6135da2fa88348d1ba752d0b88a402d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:33:48.743523 2846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6135da2fa88348d1ba752d0b88a402d4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:33:48.744347 2846 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6135da2fa88348d1ba752d0b88a402d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6135da2fa88348d1ba752d0b88a402d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37293 } }
I20260927 02:33:48.744694 2846 leader_election.cc:302] T 00000000000000000000000000000000 P 6135da2fa88348d1ba752d0b88a402d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6135da2fa88348d1ba752d0b88a402d4; no voters:
I20260927 02:33:48.745028 2846 leader_election.cc:288] T 00000000000000000000000000000000 P 6135da2fa88348d1ba752d0b88a402d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:33:48.745090 2851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6135da2fa88348d1ba752d0b88a402d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:33:48.745275 2851 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6135da2fa88348d1ba752d0b88a402d4 [term 1 LEADER]: Becoming Leader. State: Replica: 6135da2fa88348d1ba752d0b88a402d4, State: Running, Role: LEADER
I20260927 02:33:48.745608 2851 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6135da2fa88348d1ba752d0b88a402d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6135da2fa88348d1ba752d0b88a402d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37293 } }
I20260927 02:33:48.745755 2849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:48.733330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38088 (local address 127.0.98.62:37293)
0927 02:33:48.733718 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:48.733736 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:48.733776 (+ 40us) server_negotiation.cc:407] Connection header received
0927 02:33:48.734544 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:48.734565 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:48.734968 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:48.735305 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:48.735911 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:48.737359 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:48.738212 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:48.738841 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:48.738984 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:48.741437 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:48.741448 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:48.741461 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:48.741486 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:48.742966 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:48.743432 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:48.743439 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:48.743455 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:48.743580 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:48.743864 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:48.743867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:48.743868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:48.744212 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:48.744365 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:48.744635 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 02:33:48.744891 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":97,"threads_started":1}
I20260927 02:33:48.746170 2846 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6135da2fa88348d1ba752d0b88a402d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:33:48.747859 2861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6135da2fa88348d1ba752d0b88a402d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6135da2fa88348d1ba752d0b88a402d4. Latest consensus state: current_term: 1 leader_uuid: "6135da2fa88348d1ba752d0b88a402d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6135da2fa88348d1ba752d0b88a402d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37293 } } }
I20260927 02:33:48.748047 2861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6135da2fa88348d1ba752d0b88a402d4 [sys.catalog]: This master's current role is: LEADER
I20260927 02:33:48.748349 2852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6135da2fa88348d1ba752d0b88a402d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6135da2fa88348d1ba752d0b88a402d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6135da2fa88348d1ba752d0b88a402d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37293 } } }
I20260927 02:33:48.748564 2852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6135da2fa88348d1ba752d0b88a402d4 [sys.catalog]: This master's current role is: LEADER
W20260927 02:33:48.750258 2866 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6135da2fa88348d1ba752d0b88a402d4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:33:48.750319 2866 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:33:48.750425 2868 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:33:48.751250 2868 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:33:48.756177 2868 catalog_manager.cc:1347] Generated new cluster ID: 647cb3837dd54f9f848ccee65ec81e6d
I20260927 02:33:48.756237 2868 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:33:48.763412 2868 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:33:48.764309 2868 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:33:48.771291 2868 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6135da2fa88348d1ba752d0b88a402d4: Generated new TSK 0
I20260927 02:33:48.771919 2868 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:33:48.805454 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37293
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:33:48.911751 2873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:33:48.911976 2873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:33:48.912014 2873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:33:48.915865 2873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:33:48.915910 2873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:33:48.915963 2873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:33:48.920884 2873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:33:48.921730 2873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:33:48.922587 2873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:33:48.931111 2881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:48.931118 2878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:48.931129 2879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:48.931360 2873 server_base.cc:1034] running on GCE node
I20260927 02:33:48.931701 2873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:33:48.932253 2873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:33:48.933396 2873 hybrid_clock.cc:648] HybridClock initialized: now 1790476428933370 us; error 35 us; skew 500 ppm
Sep 27 02:33:48 dist-test-slave-grlk krb5kdc[2765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476428, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:33:48.935935 2873 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:33:48.936685 2873 webserver.cc:460] Webserver started at http://127.0.98.1:36305/ using document root <none> and password file <none>
I20260927 02:33:48.937206 2873 fs_manager.cc:359] Metadata directory not provided
I20260927 02:33:48.937269 2873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:33:48.937439 2873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:33:48.939262 2873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-0/data/instance:
uuid: "b9f92d0750c0477ea1179c992f85dd2b"
format_stamp: "Formatted at 2026-09-27 02:33:48 on dist-test-slave-grlk"
server_key: "f034fc89adcb3d03201223b261315bd2"
server_key_iv: "d544f69e6a3b5acebff281e4511a52ee"
server_key_version: "encryptionkey@0"
I20260927 02:33:48.939767 2873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b9f92d0750c0477ea1179c992f85dd2b"
format_stamp: "Formatted at 2026-09-27 02:33:48 on dist-test-slave-grlk"
server_key: "f034fc89adcb3d03201223b261315bd2"
server_key_iv: "d544f69e6a3b5acebff281e4511a52ee"
server_key_version: "encryptionkey@0"
I20260927 02:33:48.943192 2873 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 02:33:48.945407 2888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:48.946511 2873 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:33:48.946609 2873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-0/wal
uuid: "b9f92d0750c0477ea1179c992f85dd2b"
format_stamp: "Formatted at 2026-09-27 02:33:48 on dist-test-slave-grlk"
server_key: "f034fc89adcb3d03201223b261315bd2"
server_key_iv: "d544f69e6a3b5acebff281e4511a52ee"
server_key_version: "encryptionkey@0"
I20260927 02:33:48.946703 2873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:33:48.963888 2873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:33:48.966765 2873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:33:48.966929 2873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:33:48.967442 2873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:33:48.968183 2873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:33:48.968232 2873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:48.968277 2873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:33:48.968309 2873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:48.976851 2873 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41595
I20260927 02:33:48.976899 3001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41595 every 8 connection(s)
I20260927 02:33:48.977710 2873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:33:48.980796 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 2873
I20260927 02:33:48.980908 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:33:48.981181 392 external_mini_cluster.cc:1428] Setting key da1ed6a387e117290a3809984b1b71f8
Sep 27 02:33:48 dist-test-slave-grlk krb5kdc[2765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476428, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:33:48.991127 2849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:48.979273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37453 (local address 127.0.98.62:37293)
0927 02:33:48.979395 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:48.979398 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:48.980138 (+ 740us) server_negotiation.cc:407] Connection header received
0927 02:33:48.981682 (+ 1544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:48.981686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:48.981738 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:48.981809 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:48.982971 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:48.983497 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:48.984703 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:48.985052 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:48.985174 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:48.987549 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:48.987554 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:48.987557 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:48.987586 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:48.989167 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:48.989603 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:48.989607 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:48.989609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:48.989702 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:48.990042 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:48.990047 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:48.990048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:48.990266 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:48.990373 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:48.990836 (+ 463us) server_negotiation.cc:299] Negotiation successful
0927 02:33:48.990954 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 02:33:48.991951 3004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:48.979607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37293 (local address 127.0.98.1:37453)
0927 02:33:48.980005 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:48.980042 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:48.981486 (+ 1444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:48.981957 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:48.981964 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:48.982288 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:33:48.982815 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:48.982835 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:48.983613 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:48.983616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:48.984586 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:48.984594 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:48.985192 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:48.985195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:48.985322 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:48.985862 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:33:48.985876 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:33:48.987388 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:33:48.989273 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:48.989277 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:48.989287 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:48.989509 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:48.989793 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:48.989795 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:48.989797 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:48.989905 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:48.990356 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:33:48.990361 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:33:48.990624 (+ 263us) client_negotiation.cc:769] Sending connection context
0927 02:33:48.990778 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 02:33:48.991061 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":110,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 02:33:48.993021 3002 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37293
I20260927 02:33:48.993230 3002 heartbeater.cc:464] Registering TS with master...
I20260927 02:33:48.993667 3002 heartbeater.cc:511] Master 127.0.98.62:37293 requested a full tablet report, sending...
I20260927 02:33:48.995160 2810 ts_manager.cc:194] Registered new tserver with Master: b9f92d0750c0477ea1179c992f85dd2b (127.0.98.1:41595)
I20260927 02:33:48.996193 2810 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37453
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:33:49.007483 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37293
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:33:49.110862 3007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:33:49.111214 3007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:33:49.111279 3007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:33:49.117076 3007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:33:49.117134 3007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:33:49.117193 3007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:33:49.122372 3007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:33:49.123281 3007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:33:49.124166 3007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:33:49.132865 3015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:49.133026 3013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:49.133088 3007 server_base.cc:1034] running on GCE node
W20260927 02:33:49.133119 3012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:49.133507 3007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:33:49.134074 3007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:33:49.135288 3007 hybrid_clock.cc:648] HybridClock initialized: now 1790476429135266 us; error 27 us; skew 500 ppm
Sep 27 02:33:49 dist-test-slave-grlk krb5kdc[2765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476429, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:33:49.137959 3007 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:33:49.138765 3007 webserver.cc:460] Webserver started at http://127.0.98.2:41445/ using document root <none> and password file <none>
I20260927 02:33:49.139276 3007 fs_manager.cc:359] Metadata directory not provided
I20260927 02:33:49.139340 3007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:33:49.139520 3007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:33:49.141484 3007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-1/data/instance:
uuid: "3155faae0826411c8dc09863131346dc"
format_stamp: "Formatted at 2026-09-27 02:33:49 on dist-test-slave-grlk"
server_key: "c498ebc06b916ecc13fa47717242d705"
server_key_iv: "754181292b062e57e47ea28fba39f332"
server_key_version: "encryptionkey@0"
I20260927 02:33:49.142011 3007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3155faae0826411c8dc09863131346dc"
format_stamp: "Formatted at 2026-09-27 02:33:49 on dist-test-slave-grlk"
server_key: "c498ebc06b916ecc13fa47717242d705"
server_key_iv: "754181292b062e57e47ea28fba39f332"
server_key_version: "encryptionkey@0"
I20260927 02:33:49.145480 3007 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:33:49.147873 3022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:49.149122 3007 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:33:49.149231 3007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-1/wal
uuid: "3155faae0826411c8dc09863131346dc"
format_stamp: "Formatted at 2026-09-27 02:33:49 on dist-test-slave-grlk"
server_key: "c498ebc06b916ecc13fa47717242d705"
server_key_iv: "754181292b062e57e47ea28fba39f332"
server_key_version: "encryptionkey@0"
I20260927 02:33:49.149338 3007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:33:49.173245 3007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:33:49.176122 3007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:33:49.176281 3007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:33:49.176803 3007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:33:49.177575 3007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:33:49.177619 3007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:49.177659 3007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:33:49.177690 3007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:49.185616 3007 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34263
I20260927 02:33:49.185658 3135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34263 every 8 connection(s)
I20260927 02:33:49.186384 3007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:33:49.192673 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 3007
I20260927 02:33:49.192759 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:33:49.193001 392 external_mini_cluster.cc:1428] Setting key eeb2c1ea41bb44e639d06d5b5868fd2f
Sep 27 02:33:49 dist-test-slave-grlk krb5kdc[2765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476429, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:33:49.198909 2849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:49.187947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47771 (local address 127.0.98.62:37293)
0927 02:33:49.188071 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:49.188074 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:49.188867 (+ 793us) server_negotiation.cc:407] Connection header received
0927 02:33:49.190211 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:49.190215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:49.190269 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:49.190338 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:49.191418 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:49.191930 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:49.193110 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:49.193409 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:49.193513 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:49.195893 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:49.195897 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:49.195899 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:49.195921 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:49.197197 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:49.197625 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:49.197628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:49.197630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:49.197696 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:49.197973 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:49.197976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:49.197978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:49.198138 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:49.198217 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:49.198652 (+ 435us) server_negotiation.cc:299] Negotiation successful
0927 02:33:49.198760 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 02:33:49.199627 3138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:49.188289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37293 (local address 127.0.98.2:47771)
0927 02:33:49.188722 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:49.188761 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:49.190006 (+ 1245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:49.190509 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:49.190516 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:49.190827 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:33:49.191282 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:49.191295 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:49.192047 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:49.192050 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:49.192988 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:49.192997 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:49.193551 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:49.193554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:49.193728 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:49.194293 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:33:49.194313 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:33:49.195761 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:33:49.197295 (+ 1534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:49.197300 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:49.197314 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:49.197538 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:49.197771 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:49.197774 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:49.197776 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:49.197894 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:49.198227 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:33:49.198233 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:33:49.198448 (+ 215us) client_negotiation.cc:769] Sending connection context
0927 02:33:49.198600 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 02:33:49.198880 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":114,"threads_started":1}
I20260927 02:33:49.200701 3136 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37293
I20260927 02:33:49.200963 3136 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:33:49.201460 3136 heartbeater.cc:511] Master 127.0.98.62:37293 requested a full tablet report, sending...
I20260927 02:33:49.202330 2810 ts_manager.cc:194] Registered new tserver with Master: 3155faae0826411c8dc09863131346dc (127.0.98.2:34263)
I20260927 02:33:49.202877 2810 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47771
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:33:49.215386 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37293
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:33:49.318405 3141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:33:49.318655 3141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:33:49.318694 3141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:33:49.322623 3141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:33:49.322669 3141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:33:49.322724 3141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:33:49.327651 3141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:33:49.328527 3141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:33:49.329459 3141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:33:49.338302 3147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:49.338356 3146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:49.338356 3149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:49.338491 3141 server_base.cc:1034] running on GCE node
I20260927 02:33:49.338886 3141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:33:49.339465 3141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:33:49.340642 3141 hybrid_clock.cc:648] HybridClock initialized: now 1790476429340615 us; error 38 us; skew 500 ppm
Sep 27 02:33:49 dist-test-slave-grlk krb5kdc[2765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476429, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:33:49.343254 3141 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:33:49.344105 3141 webserver.cc:460] Webserver started at http://127.0.98.3:43133/ using document root <none> and password file <none>
I20260927 02:33:49.344606 3141 fs_manager.cc:359] Metadata directory not provided
I20260927 02:33:49.344666 3141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:33:49.344838 3141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:33:49.346796 3141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-2/data/instance:
uuid: "3d018ae14c73490789b8a68014bd18f2"
format_stamp: "Formatted at 2026-09-27 02:33:49 on dist-test-slave-grlk"
server_key: "55bab78a0f948cf8db62b4be71620286"
server_key_iv: "d9d66f70427e7c60241bd43257aa1edf"
server_key_version: "encryptionkey@0"
I20260927 02:33:49.347314 3141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3d018ae14c73490789b8a68014bd18f2"
format_stamp: "Formatted at 2026-09-27 02:33:49 on dist-test-slave-grlk"
server_key: "55bab78a0f948cf8db62b4be71620286"
server_key_iv: "d9d66f70427e7c60241bd43257aa1edf"
server_key_version: "encryptionkey@0"
I20260927 02:33:49.350803 3141 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:33:49.353034 3156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:49.354275 3141 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:33:49.354379 3141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-2/wal
uuid: "3d018ae14c73490789b8a68014bd18f2"
format_stamp: "Formatted at 2026-09-27 02:33:49 on dist-test-slave-grlk"
server_key: "55bab78a0f948cf8db62b4be71620286"
server_key_iv: "d9d66f70427e7c60241bd43257aa1edf"
server_key_version: "encryptionkey@0"
I20260927 02:33:49.354490 3141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:33:49.364379 3141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:33:49.366878 3141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:33:49.367033 3141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:33:49.367540 3141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:33:49.368255 3141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:33:49.368299 3141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:49.368346 3141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:33:49.368372 3141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:49.376819 3141 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35389
I20260927 02:33:49.377056 3269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35389 every 8 connection(s)
I20260927 02:33:49.377734 3141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:33:49.380339 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 3141
I20260927 02:33:49.380445 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:33:49.380740 392 external_mini_cluster.cc:1428] Setting key 7f909da025bea6d2f1489e945b4828ac
Sep 27 02:33:49 dist-test-slave-grlk krb5kdc[2765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476429, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:33:49.390283 2849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:49.379456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52021 (local address 127.0.98.62:37293)
0927 02:33:49.379622 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:49.379627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:49.380441 (+ 814us) server_negotiation.cc:407] Connection header received
0927 02:33:49.382097 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:49.382100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:49.382149 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:49.382217 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:49.383344 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:49.383863 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:49.384954 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:49.385168 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:49.385252 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:49.387436 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:49.387440 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:49.387442 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:49.387471 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:49.388521 (+ 1050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:49.388964 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:49.388967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:49.388969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:49.389040 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:49.389327 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:49.389330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:49.389332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:49.389501 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:49.389582 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:49.390020 (+ 438us) server_negotiation.cc:299] Negotiation successful
0927 02:33:49.390129 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 02:33:49.391002 3272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:49.379830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37293 (local address 127.0.98.3:52021)
0927 02:33:49.380283 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:49.380339 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:49.381892 (+ 1553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:49.382373 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:49.382379 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:49.382754 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:33:49.383217 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:49.383230 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:49.383979 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:49.383982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:49.384825 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:49.384834 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:49.385295 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:49.385299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:49.385462 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:49.385940 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:33:49.385954 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:33:49.387294 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:33:49.388618 (+ 1324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:49.388623 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:49.388636 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:49.388858 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:49.389117 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:49.389120 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:49.389122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:49.389246 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:49.389592 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:33:49.389598 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:33:49.389816 (+ 218us) client_negotiation.cc:769] Sending connection context
0927 02:33:49.389973 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 02:33:49.390256 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":107,"threads_started":1}
I20260927 02:33:49.391991 3270 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37293
I20260927 02:33:49.392179 3270 heartbeater.cc:464] Registering TS with master...
I20260927 02:33:49.392583 3270 heartbeater.cc:511] Master 127.0.98.62:37293 requested a full tablet report, sending...
I20260927 02:33:49.393596 2810 ts_manager.cc:194] Registered new tserver with Master: 3d018ae14c73490789b8a68014bd18f2 (127.0.98.3:35389)
I20260927 02:33:49.394218 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:33:49.394212 2810 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52021
W20260927 02:33:49.405675 2849 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:33:49.404443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38100 (local address 127.0.98.62:37293)
0927 02:33:49.404590 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:49.404593 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:49.404686 (+ 93us) server_negotiation.cc:407] Connection header received
0927 02:33:49.404791 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:49.404793 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:49.404835 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:49.404910 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:49.405573 (+ 663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38100: BlockingRecv error: recv got EOF from 127.0.0.1:38100 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 02:33:49.406737 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 2873
I20260927 02:33:49.411243 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 3007
I20260927 02:33:49.415330 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 3141
I20260927 02:33:49.419548 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 2778
2026-09-27T02:33:49Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1932 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476410933508-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:33:49 dist-test-slave-grlk krb5kdc[3282](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:33:49 dist-test-slave-grlk krb5kdc[3282](info): set up 2 sockets
Sep 27 02:33:49 dist-test-slave-grlk krb5kdc[3282](info): commencing operation
krb5kdc: starting...
W20260927 02:33:50.356443 392 mini_kdc.cc:121] Time spent starting KDC: real 0.917s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476410933508-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476410933508-392-0/krb5kdc/test-user.keytab.
Sep 27 02:33:50 dist-test-slave-grlk krb5kdc[3282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:33:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:33:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:33:50.465723 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35249
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:33:50.567793 3295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:33:50.568034 3295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:33:50.568084 3295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:33:50.572041 3295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:33:50.572088 3295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:33:50.572106 3295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:33:50.572121 3295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:33:50.572148 3295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:33:50.577167 3295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35249
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:33:50.578081 3295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:33:50.579007 3295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:33:50.586807 3301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:50.586818 3300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:50.586897 3295 server_base.cc:1034] running on GCE node
W20260927 02:33:50.586812 3303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:50.587554 3295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:33:50.588560 3295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:33:50.589744 3295 hybrid_clock.cc:648] HybridClock initialized: now 1790476430589724 us; error 30 us; skew 500 ppm
Sep 27 02:33:50 dist-test-slave-grlk krb5kdc[3282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476430, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:33:50.592161 3295 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:33:50.592968 3295 webserver.cc:460] Webserver started at http://127.0.98.62:36747/ using document root <none> and password file <none>
I20260927 02:33:50.593449 3295 fs_manager.cc:359] Metadata directory not provided
I20260927 02:33:50.593509 3295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:33:50.593698 3295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:33:50.595554 3295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "467bfc7e02d94104be56ae81ab6f1035"
format_stamp: "Formatted at 2026-09-27 02:33:50 on dist-test-slave-grlk"
server_key: "b8d4b5c8d3595e3245c42f9fb99fbfc1"
server_key_iv: "6802e7b3e21cf4c04512cd7a8c394218"
server_key_version: "encryptionkey@0"
I20260927 02:33:50.596065 3295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "467bfc7e02d94104be56ae81ab6f1035"
format_stamp: "Formatted at 2026-09-27 02:33:50 on dist-test-slave-grlk"
server_key: "b8d4b5c8d3595e3245c42f9fb99fbfc1"
server_key_iv: "6802e7b3e21cf4c04512cd7a8c394218"
server_key_version: "encryptionkey@0"
I20260927 02:33:50.599368 3295 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 02:33:50.601492 3310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:50.602633 3295 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:33:50.602739 3295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "467bfc7e02d94104be56ae81ab6f1035"
format_stamp: "Formatted at 2026-09-27 02:33:50 on dist-test-slave-grlk"
server_key: "b8d4b5c8d3595e3245c42f9fb99fbfc1"
server_key_iv: "6802e7b3e21cf4c04512cd7a8c394218"
server_key_version: "encryptionkey@0"
I20260927 02:33:50.602825 3295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:33:50.612517 3295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:33:50.615193 3295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:33:50.615350 3295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:33:50.621371 3295 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35249
I20260927 02:33:50.621604 3362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35249 every 8 connection(s)
I20260927 02:33:50.622215 3295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:33:50.624698 3363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:33:50.629789 3363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 467bfc7e02d94104be56ae81ab6f1035: Bootstrap starting.
I20260927 02:33:50.630558 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 3295
I20260927 02:33:50.630645 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:33:50.630864 392 external_mini_cluster.cc:1428] Setting key 92fe9fe2f97374186fee05b593b595eb
I20260927 02:33:50.632272 3363 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 467bfc7e02d94104be56ae81ab6f1035: Neither blocks nor log segments found. Creating new log.
I20260927 02:33:50.633121 3363 log.cc:824] T 00000000000000000000000000000000 P 467bfc7e02d94104be56ae81ab6f1035: Log is configured to *not* fsync() on all Append() calls
I20260927 02:33:50.635047 3363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 467bfc7e02d94104be56ae81ab6f1035: No bootstrap required, opened a new log
I20260927 02:33:50.638958 3363 raft_consensus.cc:348] T 00000000000000000000000000000000 P 467bfc7e02d94104be56ae81ab6f1035 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467bfc7e02d94104be56ae81ab6f1035" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35249 } }
I20260927 02:33:50.639163 3363 raft_consensus.cc:374] T 00000000000000000000000000000000 P 467bfc7e02d94104be56ae81ab6f1035 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:33:50.639197 3363 raft_consensus.cc:729] T 00000000000000000000000000000000 P 467bfc7e02d94104be56ae81ab6f1035 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 467bfc7e02d94104be56ae81ab6f1035, State: Initialized, Role: FOLLOWER
Sep 27 02:33:50 dist-test-slave-grlk krb5kdc[3282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:33:50.639608 3363 consensus_queue.cc:260] T 00000000000000000000000000000000 P 467bfc7e02d94104be56ae81ab6f1035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467bfc7e02d94104be56ae81ab6f1035" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35249 } }
I20260927 02:33:50.639732 3363 raft_consensus.cc:388] T 00000000000000000000000000000000 P 467bfc7e02d94104be56ae81ab6f1035 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:33:50.639791 3363 raft_consensus.cc:482] T 00000000000000000000000000000000 P 467bfc7e02d94104be56ae81ab6f1035 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:33:50.639884 3363 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 467bfc7e02d94104be56ae81ab6f1035 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:33:50.640899 3363 raft_consensus.cc:504] T 00000000000000000000000000000000 P 467bfc7e02d94104be56ae81ab6f1035 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467bfc7e02d94104be56ae81ab6f1035" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35249 } }
I20260927 02:33:50.641232 3363 leader_election.cc:302] T 00000000000000000000000000000000 P 467bfc7e02d94104be56ae81ab6f1035 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 467bfc7e02d94104be56ae81ab6f1035; no voters:
I20260927 02:33:50.641513 3363 leader_election.cc:288] T 00000000000000000000000000000000 P 467bfc7e02d94104be56ae81ab6f1035 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:33:50.641592 3368 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 467bfc7e02d94104be56ae81ab6f1035 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:33:50.641788 3368 raft_consensus.cc:686] T 00000000000000000000000000000000 P 467bfc7e02d94104be56ae81ab6f1035 [term 1 LEADER]: Becoming Leader. State: Replica: 467bfc7e02d94104be56ae81ab6f1035, State: Running, Role: LEADER
I20260927 02:33:50.642078 3368 consensus_queue.cc:237] T 00000000000000000000000000000000 P 467bfc7e02d94104be56ae81ab6f1035 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467bfc7e02d94104be56ae81ab6f1035" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35249 } }
I20260927 02:33:50.642788 3363 sys_catalog.cc:564] T 00000000000000000000000000000000 P 467bfc7e02d94104be56ae81ab6f1035 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:33:50.643535 3369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 467bfc7e02d94104be56ae81ab6f1035 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 467bfc7e02d94104be56ae81ab6f1035. Latest consensus state: current_term: 1 leader_uuid: "467bfc7e02d94104be56ae81ab6f1035" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467bfc7e02d94104be56ae81ab6f1035" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35249 } } }
I20260927 02:33:50.643659 3369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 467bfc7e02d94104be56ae81ab6f1035 [sys.catalog]: This master's current role is: LEADER
I20260927 02:33:50.643954 3370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 467bfc7e02d94104be56ae81ab6f1035 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "467bfc7e02d94104be56ae81ab6f1035" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467bfc7e02d94104be56ae81ab6f1035" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35249 } } }
I20260927 02:33:50.644040 3370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 467bfc7e02d94104be56ae81ab6f1035 [sys.catalog]: This master's current role is: LEADER
I20260927 02:33:50.644337 3377 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:33:50.645393 3366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:50.631849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45678 (local address 127.0.98.62:35249)
0927 02:33:50.632259 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:50.632277 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:50.632318 (+ 41us) server_negotiation.cc:407] Connection header received
0927 02:33:50.633145 (+ 827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:50.633167 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:50.633560 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:50.633867 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:50.634471 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:50.635840 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:50.636650 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:50.637367 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:50.637531 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:50.640119 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:50.640125 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:50.640141 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:50.640170 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:50.641909 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:50.642382 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:50.642387 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:50.642391 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:50.642494 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:50.642752 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:50.642755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:50.642757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:50.643238 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:50.644035 (+ 797us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:50.644176 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 02:33:50.644448 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":119,"threads_started":1}
I20260927 02:33:50.647253 3377 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:33:50.647933 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:33:50.652212 3377 catalog_manager.cc:1347] Generated new cluster ID: 5f120ddf159545198f2248161381907b
I20260927 02:33:50.652307 3377 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:33:50.660406 3377 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:33:50.661562 3377 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:33:50.664520 3366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:50.660223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60964 (local address 127.0.98.62:35249)
0927 02:33:50.660316 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:50.660320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:50.660726 (+ 406us) server_negotiation.cc:407] Connection header received
0927 02:33:50.660825 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:50.660828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:50.660870 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:50.660959 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:50.661413 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:50.661934 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:50.662709 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:50.662935 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:50.663017 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:50.663735 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:50.663739 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:50.663741 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:33:50.664126 (+ 385us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:50.664152 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:33:50.664167 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:50.664238 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:50.664327 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 02:33:50.664393 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":33}
I20260927 02:33:50.666708 3377 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 467bfc7e02d94104be56ae81ab6f1035: Generated new TSK 0
I20260927 02:33:50.667251 3377 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:33:50.672119 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 3295
2026-09-27T02:33:50Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1250 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:33:50 dist-test-slave-grlk krb5kdc[3396](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:33:50 dist-test-slave-grlk krb5kdc[3396](info): set up 2 sockets
Sep 27 02:33:50 dist-test-slave-grlk krb5kdc[3396](info): commencing operation
krb5kdc: starting...
W20260927 02:33:51.608660 392 mini_kdc.cc:121] Time spent starting KDC: real 0.919s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/krb5kdc/test-user.keytab.
Sep 27 02:33:51 dist-test-slave-grlk krb5kdc[3396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:33:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:33:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:33:51.713338 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39703
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39703
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:33:51.813808 3409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:33:51.814041 3409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:33:51.814095 3409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:33:51.818073 3409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:33:51.818125 3409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:33:51.818150 3409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:33:51.818171 3409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:33:51.818198 3409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:33:51.823189 3409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:33:51.824100 3409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:33:51.825049 3409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:33:51.832533 3415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:51.832574 3409 server_base.cc:1034] running on GCE node
W20260927 02:33:51.832530 3414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:51.832834 3417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:51.833333 3409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:33:51.834319 3409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:33:51.835467 3409 hybrid_clock.cc:648] HybridClock initialized: now 1790476431835446 us; error 35 us; skew 500 ppm
Sep 27 02:33:51 dist-test-slave-grlk krb5kdc[3396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:33:51.837893 3409 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:33:51.838653 3409 webserver.cc:460] Webserver started at http://127.0.98.62:43359/ using document root <none> and password file <none>
I20260927 02:33:51.839126 3409 fs_manager.cc:359] Metadata directory not provided
I20260927 02:33:51.839186 3409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:33:51.839346 3409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:33:51.841264 3409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "5f659628c7ca41ceaef7d54e216097d5"
format_stamp: "Formatted at 2026-09-27 02:33:51 on dist-test-slave-grlk"
server_key: "613e2a6b7e05fade34f192f586ca7e5b"
server_key_iv: "7c84547264d20b4578c6633411c30835"
server_key_version: "encryptionkey@0"
I20260927 02:33:51.841759 3409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "5f659628c7ca41ceaef7d54e216097d5"
format_stamp: "Formatted at 2026-09-27 02:33:51 on dist-test-slave-grlk"
server_key: "613e2a6b7e05fade34f192f586ca7e5b"
server_key_iv: "7c84547264d20b4578c6633411c30835"
server_key_version: "encryptionkey@0"
I20260927 02:33:51.845038 3409 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:33:51.847071 3424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:51.848161 3409 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:33:51.848263 3409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "5f659628c7ca41ceaef7d54e216097d5"
format_stamp: "Formatted at 2026-09-27 02:33:51 on dist-test-slave-grlk"
server_key: "613e2a6b7e05fade34f192f586ca7e5b"
server_key_iv: "7c84547264d20b4578c6633411c30835"
server_key_version: "encryptionkey@0"
I20260927 02:33:51.848343 3409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:33:51.877081 3409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:33:51.879905 3409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:33:51.880074 3409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:33:51.886142 3409 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39703
I20260927 02:33:51.886376 3476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39703 every 8 connection(s)
I20260927 02:33:51.887002 3409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:33:51.887796 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 3409
I20260927 02:33:51.887890 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:33:51.888136 392 external_mini_cluster.cc:1428] Setting key 4b140041542fd0f41edbb8dface05471
I20260927 02:33:51.889801 3477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:33:51.894744 3477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f659628c7ca41ceaef7d54e216097d5: Bootstrap starting.
Sep 27 02:33:51 dist-test-slave-grlk krb5kdc[3396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:33:51.896708 3477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f659628c7ca41ceaef7d54e216097d5: Neither blocks nor log segments found. Creating new log.
I20260927 02:33:51.897377 3477 log.cc:824] T 00000000000000000000000000000000 P 5f659628c7ca41ceaef7d54e216097d5: Log is configured to *not* fsync() on all Append() calls
I20260927 02:33:51.899232 3477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f659628c7ca41ceaef7d54e216097d5: No bootstrap required, opened a new log
I20260927 02:33:51.900390 3480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:51.889325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37846 (local address 127.0.98.62:39703)
0927 02:33:51.889724 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:51.889738 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:51.889771 (+ 33us) server_negotiation.cc:407] Connection header received
0927 02:33:51.890418 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:51.890434 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:51.890738 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:51.891017 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:51.891737 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:51.892680 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:51.893532 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:51.893980 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:51.894116 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:51.896262 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:51.896267 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:51.896278 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:51.896301 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:51.897887 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:51.898360 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:51.898364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:51.898369 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:51.898480 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:51.898740 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:51.898742 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:51.898743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:51.899083 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:51.899209 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:51.899497 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 02:33:51.899749 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":139,"threads_started":1}
I20260927 02:33:51.902457 3477 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f659628c7ca41ceaef7d54e216097d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f659628c7ca41ceaef7d54e216097d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39703 } }
I20260927 02:33:51.902647 3477 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f659628c7ca41ceaef7d54e216097d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:33:51.902693 3477 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f659628c7ca41ceaef7d54e216097d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f659628c7ca41ceaef7d54e216097d5, State: Initialized, Role: FOLLOWER
I20260927 02:33:51.903053 3477 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f659628c7ca41ceaef7d54e216097d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f659628c7ca41ceaef7d54e216097d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39703 } }
I20260927 02:33:51.903163 3477 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f659628c7ca41ceaef7d54e216097d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:33:51.903219 3477 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f659628c7ca41ceaef7d54e216097d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:33:51.903308 3477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f659628c7ca41ceaef7d54e216097d5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:33:51.904062 3477 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f659628c7ca41ceaef7d54e216097d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f659628c7ca41ceaef7d54e216097d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39703 } }
I20260927 02:33:51.904341 3477 leader_election.cc:302] T 00000000000000000000000000000000 P 5f659628c7ca41ceaef7d54e216097d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f659628c7ca41ceaef7d54e216097d5; no voters:
I20260927 02:33:51.904574 3477 leader_election.cc:288] T 00000000000000000000000000000000 P 5f659628c7ca41ceaef7d54e216097d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:33:51.904670 3482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f659628c7ca41ceaef7d54e216097d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:33:51.904875 3482 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f659628c7ca41ceaef7d54e216097d5 [term 1 LEADER]: Becoming Leader. State: Replica: 5f659628c7ca41ceaef7d54e216097d5, State: Running, Role: LEADER
I20260927 02:33:51.905175 3482 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f659628c7ca41ceaef7d54e216097d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f659628c7ca41ceaef7d54e216097d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39703 } }
I20260927 02:33:51.905601 3477 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f659628c7ca41ceaef7d54e216097d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:33:51.906839 3484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f659628c7ca41ceaef7d54e216097d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f659628c7ca41ceaef7d54e216097d5. Latest consensus state: current_term: 1 leader_uuid: "5f659628c7ca41ceaef7d54e216097d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f659628c7ca41ceaef7d54e216097d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39703 } } }
I20260927 02:33:51.907152 3484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f659628c7ca41ceaef7d54e216097d5 [sys.catalog]: This master's current role is: LEADER
I20260927 02:33:51.907053 3483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f659628c7ca41ceaef7d54e216097d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f659628c7ca41ceaef7d54e216097d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f659628c7ca41ceaef7d54e216097d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39703 } } }
I20260927 02:33:51.907333 3483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f659628c7ca41ceaef7d54e216097d5 [sys.catalog]: This master's current role is: LEADER
I20260927 02:33:51.907702 3496 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:33:51.910207 3496 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:33:51.914664 3496 catalog_manager.cc:1347] Generated new cluster ID: 591f2476f3dc4920b3134f6febb2545c
I20260927 02:33:51.914721 3496 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:33:51.944525 3496 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:33:51.945467 3496 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:33:51.952723 3496 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f659628c7ca41ceaef7d54e216097d5: Generated new TSK 0
I20260927 02:33:51.953454 3496 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:33:51.985441 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39703
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:33:52.087944 3503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:33:52.088177 3503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:33:52.088217 3503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:33:52.092135 3503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:33:52.092183 3503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:33:52.092242 3503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:33:52.097157 3503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:33:52.098003 3503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:33:52.098883 3503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:33:52.107424 3511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:52.107400 3509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:52.107412 3508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:52.107650 3503 server_base.cc:1034] running on GCE node
I20260927 02:33:52.108065 3503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:33:52.108652 3503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:33:52.109802 3503 hybrid_clock.cc:648] HybridClock initialized: now 1790476432109783 us; error 29 us; skew 500 ppm
Sep 27 02:33:52 dist-test-slave-grlk krb5kdc[3396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:33:52.112228 3503 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:33:52.113016 3503 webserver.cc:460] Webserver started at http://127.0.98.1:43191/ using document root <none> and password file <none>
I20260927 02:33:52.113507 3503 fs_manager.cc:359] Metadata directory not provided
I20260927 02:33:52.113561 3503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:33:52.113751 3503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:33:52.115674 3503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-0/data/instance:
uuid: "902573fb873c46459a0bd58a78097c02"
format_stamp: "Formatted at 2026-09-27 02:33:52 on dist-test-slave-grlk"
server_key: "5a751d16bd15ca1e713d96a3648fbddd"
server_key_iv: "7ee40968ad66504a29732f34c732acf0"
server_key_version: "encryptionkey@0"
I20260927 02:33:52.116168 3503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-0/wal/instance:
uuid: "902573fb873c46459a0bd58a78097c02"
format_stamp: "Formatted at 2026-09-27 02:33:52 on dist-test-slave-grlk"
server_key: "5a751d16bd15ca1e713d96a3648fbddd"
server_key_iv: "7ee40968ad66504a29732f34c732acf0"
server_key_version: "encryptionkey@0"
I20260927 02:33:52.119488 3503 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 02:33:52.121634 3518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:52.122726 3503 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:33:52.122822 3503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-0/wal
uuid: "902573fb873c46459a0bd58a78097c02"
format_stamp: "Formatted at 2026-09-27 02:33:52 on dist-test-slave-grlk"
server_key: "5a751d16bd15ca1e713d96a3648fbddd"
server_key_iv: "7ee40968ad66504a29732f34c732acf0"
server_key_version: "encryptionkey@0"
I20260927 02:33:52.122912 3503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:33:52.143131 3503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:33:52.145812 3503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:33:52.145974 3503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:33:52.146453 3503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:33:52.147166 3503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:33:52.147209 3503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:52.147244 3503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:33:52.147264 3503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:52.154942 3503 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45937
I20260927 02:33:52.155018 3631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45937 every 8 connection(s)
I20260927 02:33:52.155747 3503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:33:52.160238 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 3503
I20260927 02:33:52.160326 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:33:52.160564 392 external_mini_cluster.cc:1428] Setting key 705f373c973fe0345b17bc894ea597f7
Sep 27 02:33:52 dist-test-slave-grlk krb5kdc[3396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:33:52.168454 3480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:52.157273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56339 (local address 127.0.98.62:39703)
0927 02:33:52.157378 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:52.157382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:52.158099 (+ 717us) server_negotiation.cc:407] Connection header received
0927 02:33:52.159375 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:52.159379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:52.159434 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:52.159505 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:52.160623 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:52.161294 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:52.162448 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:52.162684 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:52.162769 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:52.165248 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:52.165251 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:52.165254 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:52.165275 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:52.166564 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:52.167088 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:52.167090 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:52.167091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:52.167154 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:52.167475 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:52.167477 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:52.167478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:52.167622 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:52.167688 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:52.168243 (+ 555us) server_negotiation.cc:299] Negotiation successful
0927 02:33:52.168340 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:33:52.169459 3634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:52.157591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.1:56339)
0927 02:33:52.157970 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:52.158008 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:52.159185 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:52.159671 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:52.159678 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:52.159993 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:33:52.160491 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:52.160507 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:52.161412 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:52.161415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:52.162342 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:52.162351 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:52.162809 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:52.162813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:52.162999 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:52.163525 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:33:52.163542 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:33:52.165100 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:33:52.166674 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:52.166679 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:52.166693 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:52.167006 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:52.167226 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:52.167229 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:52.167231 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:52.167397 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:52.167706 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:33:52.167712 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:33:52.167996 (+ 284us) client_negotiation.cc:769] Sending connection context
0927 02:33:52.168199 (+ 203us) client_negotiation.cc:240] Negotiation successful
0927 02:33:52.168520 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":101,"threads_started":1}
I20260927 02:33:52.170502 3632 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39703
I20260927 02:33:52.170715 3632 heartbeater.cc:464] Registering TS with master...
I20260927 02:33:52.171116 3632 heartbeater.cc:511] Master 127.0.98.62:39703 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 02:33:52.172487 3441 ts_manager.cc:194] Registered new tserver with Master: 902573fb873c46459a0bd58a78097c02 (127.0.98.1:45937)
I20260927 02:33:52.173862 3441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56339
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:33:52.183162 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39703
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:33:52.284492 3637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:33:52.284710 3637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:33:52.284749 3637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:33:52.288516 3637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:33:52.288561 3637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:33:52.288614 3637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:33:52.293460 3637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:33:52.294377 3637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:33:52.295279 3637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:33:52.303936 3642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:52.303923 3643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:52.304102 3637 server_base.cc:1034] running on GCE node
W20260927 02:33:52.303932 3645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:52.304593 3637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:33:52.305228 3637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:33:52.306375 3637 hybrid_clock.cc:648] HybridClock initialized: now 1790476432306358 us; error 32 us; skew 500 ppm
Sep 27 02:33:52 dist-test-slave-grlk krb5kdc[3396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:33:52.308828 3637 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:33:52.309610 3637 webserver.cc:460] Webserver started at http://127.0.98.2:44463/ using document root <none> and password file <none>
I20260927 02:33:52.310104 3637 fs_manager.cc:359] Metadata directory not provided
I20260927 02:33:52.310160 3637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:33:52.310338 3637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:33:52.312170 3637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-1/data/instance:
uuid: "0daaac4d1c374391a7bd18e63c3fa175"
format_stamp: "Formatted at 2026-09-27 02:33:52 on dist-test-slave-grlk"
server_key: "664b1cf783e9525d4f37589372221f47"
server_key_iv: "25d52c6cf5b77f0e0a3407f249e76cd6"
server_key_version: "encryptionkey@0"
I20260927 02:33:52.312655 3637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0daaac4d1c374391a7bd18e63c3fa175"
format_stamp: "Formatted at 2026-09-27 02:33:52 on dist-test-slave-grlk"
server_key: "664b1cf783e9525d4f37589372221f47"
server_key_iv: "25d52c6cf5b77f0e0a3407f249e76cd6"
server_key_version: "encryptionkey@0"
I20260927 02:33:52.315982 3637 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 02:33:52.318168 3652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:52.319212 3637 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:33:52.319300 3637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-1/wal
uuid: "0daaac4d1c374391a7bd18e63c3fa175"
format_stamp: "Formatted at 2026-09-27 02:33:52 on dist-test-slave-grlk"
server_key: "664b1cf783e9525d4f37589372221f47"
server_key_iv: "25d52c6cf5b77f0e0a3407f249e76cd6"
server_key_version: "encryptionkey@0"
I20260927 02:33:52.319389 3637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:33:52.331663 3637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:33:52.334370 3637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:33:52.334517 3637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:33:52.334992 3637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:33:52.335690 3637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:33:52.335731 3637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:52.335768 3637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:33:52.335795 3637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:52.343340 3637 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34521
I20260927 02:33:52.343372 3765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34521 every 8 connection(s)
I20260927 02:33:52.344086 3637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:33:52.347766 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 3637
I20260927 02:33:52.347882 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:33:52.348129 392 external_mini_cluster.cc:1428] Setting key 4c6136dda9c37877651d72b95808356d
Sep 27 02:33:52 dist-test-slave-grlk krb5kdc[3396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:33:52.356709 3480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:52.345608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44663 (local address 127.0.98.62:39703)
0927 02:33:52.345751 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:52.345756 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:52.346482 (+ 726us) server_negotiation.cc:407] Connection header received
0927 02:33:52.347728 (+ 1246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:52.347731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:52.347777 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:52.347866 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:52.349025 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:52.349648 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:52.350737 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:52.350960 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:52.351043 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:52.353554 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:52.353557 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:52.353559 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:52.353581 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:52.354813 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:52.355347 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:52.355350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:52.355351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:52.355414 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:52.355736 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:52.355738 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:52.355739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:52.355882 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:52.355947 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:52.356505 (+ 558us) server_negotiation.cc:299] Negotiation successful
0927 02:33:52.356594 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:33:52.357625 3768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:52.345945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.2:44663)
0927 02:33:52.346352 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:52.346389 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:52.347518 (+ 1129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:52.348026 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:52.348032 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:52.348371 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:33:52.348872 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:52.348885 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:52.349760 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:52.349763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:52.350630 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:52.350638 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:52.351094 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:52.351098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:52.351264 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:52.351794 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:33:52.351813 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:33:52.353397 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:33:52.354920 (+ 1523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:52.354926 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:52.354941 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:52.355265 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:52.355487 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:52.355490 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:52.355492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:52.355659 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:52.355972 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:33:52.355978 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:33:52.356279 (+ 301us) client_negotiation.cc:769] Sending connection context
0927 02:33:52.356487 (+ 208us) client_negotiation.cc:240] Negotiation successful
0927 02:33:52.356790 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":125,"threads_started":1}
I20260927 02:33:52.358578 3766 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39703
I20260927 02:33:52.358779 3766 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 02:33:52.359289 3766 heartbeater.cc:511] Master 127.0.98.62:39703 requested a full tablet report, sending...
I20260927 02:33:52.360235 3441 ts_manager.cc:194] Registered new tserver with Master: 0daaac4d1c374391a7bd18e63c3fa175 (127.0.98.2:34521)
I20260927 02:33:52.360773 3441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44663
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:33:52.371856 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39703
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:33:52.477311 3771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:33:52.477560 3771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:33:52.477612 3771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:33:52.481556 3771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:33:52.481611 3771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:33:52.481678 3771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:33:52.486896 3771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:33:52.487907 3771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:33:52.488845 3771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:33:52.497633 3779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:52.497699 3771 server_base.cc:1034] running on GCE node
W20260927 02:33:52.497604 3777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:52.497642 3776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:52.498152 3771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:33:52.498694 3771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:33:52.499840 3771 hybrid_clock.cc:648] HybridClock initialized: now 1790476432499826 us; error 25 us; skew 500 ppm
Sep 27 02:33:52 dist-test-slave-grlk krb5kdc[3396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:33:52.502344 3771 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:33:52.503113 3771 webserver.cc:460] Webserver started at http://127.0.98.3:35133/ using document root <none> and password file <none>
I20260927 02:33:52.503605 3771 fs_manager.cc:359] Metadata directory not provided
I20260927 02:33:52.503660 3771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:33:52.503839 3771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:33:52.505795 3771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-2/data/instance:
uuid: "b29f080d5f2f4093b87033f62b9152ca"
format_stamp: "Formatted at 2026-09-27 02:33:52 on dist-test-slave-grlk"
server_key: "24ef86ab6a406cac19104920af780ba0"
server_key_iv: "ed95b1f710b3e7bee835f3b43c455b1f"
server_key_version: "encryptionkey@0"
I20260927 02:33:52.506304 3771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b29f080d5f2f4093b87033f62b9152ca"
format_stamp: "Formatted at 2026-09-27 02:33:52 on dist-test-slave-grlk"
server_key: "24ef86ab6a406cac19104920af780ba0"
server_key_iv: "ed95b1f710b3e7bee835f3b43c455b1f"
server_key_version: "encryptionkey@0"
I20260927 02:33:52.509732 3771 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:33:52.511853 3786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:52.512989 3771 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:33:52.513080 3771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-2/wal
uuid: "b29f080d5f2f4093b87033f62b9152ca"
format_stamp: "Formatted at 2026-09-27 02:33:52 on dist-test-slave-grlk"
server_key: "24ef86ab6a406cac19104920af780ba0"
server_key_iv: "ed95b1f710b3e7bee835f3b43c455b1f"
server_key_version: "encryptionkey@0"
I20260927 02:33:52.513166 3771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:33:52.531965 3771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:33:52.534665 3771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:33:52.534817 3771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:33:52.535365 3771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:33:52.536116 3771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:33:52.536159 3771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:52.536199 3771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:33:52.536228 3771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:52.544199 3771 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34941
I20260927 02:33:52.544227 3899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34941 every 8 connection(s)
I20260927 02:33:52.545017 3771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:33:52.547560 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 3771
I20260927 02:33:52.547657 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476410933508-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:33:52.547894 392 external_mini_cluster.cc:1428] Setting key 0ec5ac81406a4686333a630a8552218a
Sep 27 02:33:52 dist-test-slave-grlk krb5kdc[3396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:33:52.556979 3480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:52.546537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33477 (local address 127.0.98.62:39703)
0927 02:33:52.546660 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:52.546664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:52.547390 (+ 726us) server_negotiation.cc:407] Connection header received
0927 02:33:52.548800 (+ 1410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:52.548803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:52.548855 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:52.548925 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:52.549993 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:52.550503 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:52.551562 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:52.551796 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:52.551881 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:52.554021 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:52.554024 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:52.554027 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:52.554050 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:52.555233 (+ 1183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:52.555656 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:52.555660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:52.555662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:52.555731 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:52.556009 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:52.556012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:52.556014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:52.556180 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:52.556261 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:52.556694 (+ 433us) server_negotiation.cc:299] Negotiation successful
0927 02:33:52.556801 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 02:33:52.557668 3902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:52.546869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39703 (local address 127.0.98.3:33477)
0927 02:33:52.547261 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:52.547298 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:52.548597 (+ 1299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:52.549104 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:52.549111 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:52.549425 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:33:52.549862 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:52.549874 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:52.550609 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:52.550611 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:52.551446 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:52.551455 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:52.551920 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:52.551923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:52.552071 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:52.552562 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:33:52.552576 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:33:52.553883 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:33:52.555329 (+ 1446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:52.555334 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:52.555347 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:52.555571 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:52.555807 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:52.555810 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:52.555812 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:52.555930 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:52.556268 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:33:52.556274 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:33:52.556485 (+ 211us) client_negotiation.cc:769] Sending connection context
0927 02:33:52.556643 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 02:33:52.556920 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":106,"threads_started":1}
I20260927 02:33:52.558557 3900 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39703
I20260927 02:33:52.558736 3900 heartbeater.cc:464] Registering TS with master...
I20260927 02:33:52.559114 3900 heartbeater.cc:511] Master 127.0.98.62:39703 requested a full tablet report, sending...
I20260927 02:33:52.559940 3441 ts_manager.cc:194] Registered new tserver with Master: b29f080d5f2f4093b87033f62b9152ca (127.0.98.3:34941)
I20260927 02:33:52.560585 3441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33477
I20260927 02:33:52.561079 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 02:33:52 dist-test-slave-grlk krb5kdc[3396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476432, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 02:33:52 dist-test-slave-grlk krb5kdc[3396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476432, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:33:52.582674 3480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:52.575620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37852 (local address 127.0.98.62:39703)
0927 02:33:52.575727 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:52.575731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:52.575894 (+ 163us) server_negotiation.cc:407] Connection header received
0927 02:33:52.575964 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:52.575967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:52.576008 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:52.576077 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:52.576688 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:52.577257 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:52.578015 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:52.578210 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:52.578271 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:52.580275 (+ 2004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:52.580278 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:52.580281 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:52.580302 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:52.581499 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:52.581792 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:52.581794 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:52.581795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:52.581855 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:52.582012 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:52.582014 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:52.582015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:52.582182 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:52.582267 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:52.582469 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 02:33:52.582562 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260927 02:33:52.583022 3440 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:37852
I20260927 02:33:52.583956 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 3503
I20260927 02:33:52.588312 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 3637
I20260927 02:33:52.592559 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 3771
I20260927 02:33:52.596599 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 3409
2026-09-27T02:33:52Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1926 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:33:52 dist-test-slave-grlk krb5kdc[3912](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:33:52 dist-test-slave-grlk krb5kdc[3912](info): set up 2 sockets
Sep 27 02:33:52 dist-test-slave-grlk krb5kdc[3912](info): commencing operation
krb5kdc: starting...
W20260927 02:33:53.544656 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/krb5kdc/test-user.keytab.
Sep 27 02:33:53 dist-test-slave-grlk krb5kdc[3912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:33:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:33:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:33:53.650898 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33543
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33543
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:33:53.753634 3925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:33:53.753886 3925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:33:53.753922 3925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:33:53.757859 3925 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:33:53.757910 3925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:33:53.757927 3925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:33:53.757941 3925 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:33:53.757956 3925 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:33:53.762993 3925 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:33:53.763911 3925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:33:53.764863 3925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:33:53.772403 3931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:53.772423 3933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:53.772496 3925 server_base.cc:1034] running on GCE node
W20260927 02:33:53.772620 3930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:53.773165 3925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:33:53.774199 3925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:33:53.775396 3925 hybrid_clock.cc:648] HybridClock initialized: now 1790476433775379 us; error 32 us; skew 500 ppm
Sep 27 02:33:53 dist-test-slave-grlk krb5kdc[3912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:33:53.780527 3925 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:33:53.781437 3925 webserver.cc:460] Webserver started at http://127.0.98.62:36067/ using document root <none> and password file <none>
I20260927 02:33:53.781961 3925 fs_manager.cc:359] Metadata directory not provided
I20260927 02:33:53.782023 3925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:33:53.782212 3925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:33:53.784121 3925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "91642d5eb0f74d17884f535debd50a77"
format_stamp: "Formatted at 2026-09-27 02:33:53 on dist-test-slave-grlk"
server_key: "7137a94cbfa2e351e84ca275dd6b5fba"
server_key_iv: "dce20a80b9b85a762b702ec6397f61bc"
server_key_version: "encryptionkey@0"
I20260927 02:33:53.784623 3925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "91642d5eb0f74d17884f535debd50a77"
format_stamp: "Formatted at 2026-09-27 02:33:53 on dist-test-slave-grlk"
server_key: "7137a94cbfa2e351e84ca275dd6b5fba"
server_key_iv: "dce20a80b9b85a762b702ec6397f61bc"
server_key_version: "encryptionkey@0"
I20260927 02:33:53.787983 3925 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 02:33:53.790156 3940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:53.791277 3925 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:33:53.791391 3925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "91642d5eb0f74d17884f535debd50a77"
format_stamp: "Formatted at 2026-09-27 02:33:53 on dist-test-slave-grlk"
server_key: "7137a94cbfa2e351e84ca275dd6b5fba"
server_key_iv: "dce20a80b9b85a762b702ec6397f61bc"
server_key_version: "encryptionkey@0"
I20260927 02:33:53.791476 3925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:33:53.806066 3925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:33:53.808838 3925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:33:53.809023 3925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:33:53.815265 3925 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33543
I20260927 02:33:53.815327 3992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33543 every 8 connection(s)
I20260927 02:33:53.816146 3925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:33:53.818718 3993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:33:53.823740 3993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91642d5eb0f74d17884f535debd50a77: Bootstrap starting.
I20260927 02:33:53.825663 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 3925
I20260927 02:33:53.825771 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:33:53.826036 392 external_mini_cluster.cc:1428] Setting key 5b1d83669588c97bc266885ff7417590
I20260927 02:33:53.826195 3993 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91642d5eb0f74d17884f535debd50a77: Neither blocks nor log segments found. Creating new log.
I20260927 02:33:53.826987 3993 log.cc:824] T 00000000000000000000000000000000 P 91642d5eb0f74d17884f535debd50a77: Log is configured to *not* fsync() on all Append() calls
I20260927 02:33:53.828678 3993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91642d5eb0f74d17884f535debd50a77: No bootstrap required, opened a new log
I20260927 02:33:53.832470 3993 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91642d5eb0f74d17884f535debd50a77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91642d5eb0f74d17884f535debd50a77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33543 } }
I20260927 02:33:53.832722 3993 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91642d5eb0f74d17884f535debd50a77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:33:53.832785 3993 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91642d5eb0f74d17884f535debd50a77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91642d5eb0f74d17884f535debd50a77, State: Initialized, Role: FOLLOWER
I20260927 02:33:53.833276 3993 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91642d5eb0f74d17884f535debd50a77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91642d5eb0f74d17884f535debd50a77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33543 } }
I20260927 02:33:53.833423 3993 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91642d5eb0f74d17884f535debd50a77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:33:53.833489 3993 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91642d5eb0f74d17884f535debd50a77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:33:53.833585 3993 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91642d5eb0f74d17884f535debd50a77 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 02:33:53 dist-test-slave-grlk krb5kdc[3912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:33:53.834411 3993 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91642d5eb0f74d17884f535debd50a77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91642d5eb0f74d17884f535debd50a77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33543 } }
I20260927 02:33:53.834714 3993 leader_election.cc:302] T 00000000000000000000000000000000 P 91642d5eb0f74d17884f535debd50a77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91642d5eb0f74d17884f535debd50a77; no voters:
I20260927 02:33:53.834995 3993 leader_election.cc:288] T 00000000000000000000000000000000 P 91642d5eb0f74d17884f535debd50a77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:33:53.835086 3998 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91642d5eb0f74d17884f535debd50a77 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:33:53.835290 3998 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91642d5eb0f74d17884f535debd50a77 [term 1 LEADER]: Becoming Leader. State: Replica: 91642d5eb0f74d17884f535debd50a77, State: Running, Role: LEADER
I20260927 02:33:53.835626 3998 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91642d5eb0f74d17884f535debd50a77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91642d5eb0f74d17884f535debd50a77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33543 } }
I20260927 02:33:53.836088 3993 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91642d5eb0f74d17884f535debd50a77 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:33:53.837203 3999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91642d5eb0f74d17884f535debd50a77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91642d5eb0f74d17884f535debd50a77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91642d5eb0f74d17884f535debd50a77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33543 } } }
I20260927 02:33:53.837296 3999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91642d5eb0f74d17884f535debd50a77 [sys.catalog]: This master's current role is: LEADER
I20260927 02:33:53.837651 4000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91642d5eb0f74d17884f535debd50a77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91642d5eb0f74d17884f535debd50a77. Latest consensus state: current_term: 1 leader_uuid: "91642d5eb0f74d17884f535debd50a77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91642d5eb0f74d17884f535debd50a77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33543 } } }
I20260927 02:33:53.837745 4000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91642d5eb0f74d17884f535debd50a77 [sys.catalog]: This master's current role is: LEADER
I20260927 02:33:53.839784 3996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:53.827021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37618 (local address 127.0.98.62:33543)
0927 02:33:53.827394 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:53.827408 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:53.827441 (+ 33us) server_negotiation.cc:407] Connection header received
0927 02:33:53.828039 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:53.828055 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:53.828350 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:53.828609 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:53.829413 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:53.830374 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:53.831505 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:53.831931 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:53.832043 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:53.834553 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:53.834559 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:53.834574 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:53.834601 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:53.836585 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:53.837074 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:53.837079 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:53.837085 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:53.837217 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:53.837540 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:53.837543 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:53.837545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:53.838035 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:53.838211 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:53.838615 (+ 404us) server_negotiation.cc:299] Negotiation successful
0927 02:33:53.838889 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":113,"threads_started":1}
W20260927 02:33:53.840605 4013 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 91642d5eb0f74d17884f535debd50a77: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:33:53.840677 4013 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:33:53.840740 4008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:33:53.841547 4008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:33:53.846536 4008 catalog_manager.cc:1347] Generated new cluster ID: a8283565e5144dea969396e5f339573f
I20260927 02:33:53.846585 4008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:33:53.860205 4008 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:33:53.861019 4008 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:33:53.867703 4008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91642d5eb0f74d17884f535debd50a77: Generated new TSK 0
I20260927 02:33:53.868326 4008 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:33:53.899384 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33543
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:33:54.023039 4019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:33:54.023252 4019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:33:54.023289 4019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:33:54.027060 4019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:33:54.027099 4019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:33:54.027150 4019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:33:54.031893 4019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:33:54.032706 4019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:33:54.033584 4019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:33:54.042084 4024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:54.042083 4027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:54.042078 4025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:54.042207 4019 server_base.cc:1034] running on GCE node
I20260927 02:33:54.042703 4019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:33:54.043264 4019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:33:54.044453 4019 hybrid_clock.cc:648] HybridClock initialized: now 1790476434044430 us; error 32 us; skew 500 ppm
Sep 27 02:33:54 dist-test-slave-grlk krb5kdc[3912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:33:54.047026 4019 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:33:54.047750 4019 webserver.cc:460] Webserver started at http://127.0.98.1:43413/ using document root <none> and password file <none>
I20260927 02:33:54.048214 4019 fs_manager.cc:359] Metadata directory not provided
I20260927 02:33:54.048264 4019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:33:54.048413 4019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:33:54.050237 4019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/ts-0/data/instance:
uuid: "17e9c8df18ec47c29ef4271cda138754"
format_stamp: "Formatted at 2026-09-27 02:33:54 on dist-test-slave-grlk"
server_key: "db06ec82799eb339b6eb1aad017c3db7"
server_key_iv: "f4538243e01b6622941f940dc6869100"
server_key_version: "encryptionkey@0"
I20260927 02:33:54.050698 4019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/ts-0/wal/instance:
uuid: "17e9c8df18ec47c29ef4271cda138754"
format_stamp: "Formatted at 2026-09-27 02:33:54 on dist-test-slave-grlk"
server_key: "db06ec82799eb339b6eb1aad017c3db7"
server_key_iv: "f4538243e01b6622941f940dc6869100"
server_key_version: "encryptionkey@0"
I20260927 02:33:54.054016 4019 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260927 02:33:54.056272 4034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:54.057454 4019 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:33:54.057549 4019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/ts-0/wal
uuid: "17e9c8df18ec47c29ef4271cda138754"
format_stamp: "Formatted at 2026-09-27 02:33:54 on dist-test-slave-grlk"
server_key: "db06ec82799eb339b6eb1aad017c3db7"
server_key_iv: "f4538243e01b6622941f940dc6869100"
server_key_version: "encryptionkey@0"
I20260927 02:33:54.057648 4019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:33:54.106266 4019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:33:54.108883 4019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:33:54.109061 4019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:33:54.109591 4019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:33:54.110328 4019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:33:54.110373 4019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:54.110411 4019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:33:54.110425 4019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:54.118393 4019 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35241
I20260927 02:33:54.118420 4147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35241 every 8 connection(s)
I20260927 02:33:54.119197 4019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:33:54.125501 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 4019
I20260927 02:33:54.125597 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:33:54.125830 392 external_mini_cluster.cc:1428] Setting key f12cc6a853b499139cc130872b56179d
Sep 27 02:33:54 dist-test-slave-grlk krb5kdc[3912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:33:54.131845 3996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:54.120760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59457 (local address 127.0.98.62:33543)
0927 02:33:54.120909 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:54.120913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:54.121735 (+ 822us) server_negotiation.cc:407] Connection header received
0927 02:33:54.122990 (+ 1255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:54.122993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:54.123037 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:54.123106 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:54.124217 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:54.124753 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:54.125955 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:54.126184 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:54.126275 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:54.128576 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:54.128579 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:54.128581 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:54.128602 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:54.129934 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:54.130463 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:54.130466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:54.130467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:54.130535 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:54.130859 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:54.130861 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:54.130863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:54.131013 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:54.131082 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:54.131633 (+ 551us) server_negotiation.cc:299] Negotiation successful
0927 02:33:54.131730 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 02:33:54.132997 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:54.121164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33543 (local address 127.0.98.1:59457)
0927 02:33:54.121595 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:54.121635 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:54.122810 (+ 1175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:54.123260 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:54.123266 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:54.123603 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:33:54.124088 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:54.124102 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:54.124867 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:54.124871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:54.125837 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:54.125846 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:54.126320 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:54.126324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:54.126503 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:54.127096 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:33:54.127116 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:33:54.128429 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:33:54.130042 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:54.130048 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:54.130062 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:54.130377 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:54.130614 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:54.130617 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:54.130619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:54.130783 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:54.131106 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:33:54.131112 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:33:54.131412 (+ 300us) client_negotiation.cc:769] Sending connection context
0927 02:33:54.131616 (+ 204us) client_negotiation.cc:240] Negotiation successful
0927 02:33:54.131948 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":95,"threads_started":1}
I20260927 02:33:54.134224 4148 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33543
I20260927 02:33:54.134434 4148 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:33:54.134886 4148 heartbeater.cc:511] Master 127.0.98.62:33543 requested a full tablet report, sending...
I20260927 02:33:54.136312 3957 ts_manager.cc:194] Registered new tserver with Master: 17e9c8df18ec47c29ef4271cda138754 (127.0.98.1:35241)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 02:33:54.137573 3957 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59457
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:33:54.149832 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33543
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:33:54.252427 4153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:33:54.252636 4153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:33:54.252674 4153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:33:54.256886 4153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:33:54.256965 4153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:33:54.257035 4153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:33:54.261893 4153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:33:54.262717 4153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:33:54.263557 4153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:33:54.272162 4159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:54.272198 4153 server_base.cc:1034] running on GCE node
W20260927 02:33:54.272235 4158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:54.272336 4161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:54.272670 4153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:33:54.273267 4153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:33:54.274413 4153 hybrid_clock.cc:648] HybridClock initialized: now 1790476434274395 us; error 29 us; skew 500 ppm
Sep 27 02:33:54 dist-test-slave-grlk krb5kdc[3912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:33:54.276741 4153 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:33:54.277500 4153 webserver.cc:460] Webserver started at http://127.0.98.2:36795/ using document root <none> and password file <none>
I20260927 02:33:54.277961 4153 fs_manager.cc:359] Metadata directory not provided
I20260927 02:33:54.278013 4153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:33:54.278174 4153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:33:54.279969 4153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/ts-1/data/instance:
uuid: "0118eeda11094ec9bedcfb87a9fd5df1"
format_stamp: "Formatted at 2026-09-27 02:33:54 on dist-test-slave-grlk"
server_key: "e4289e52c67443b64796ffc7b4f847e5"
server_key_iv: "86cfc166b7acc4473ad9d065c32639e0"
server_key_version: "encryptionkey@0"
I20260927 02:33:54.280434 4153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0118eeda11094ec9bedcfb87a9fd5df1"
format_stamp: "Formatted at 2026-09-27 02:33:54 on dist-test-slave-grlk"
server_key: "e4289e52c67443b64796ffc7b4f847e5"
server_key_iv: "86cfc166b7acc4473ad9d065c32639e0"
server_key_version: "encryptionkey@0"
I20260927 02:33:54.283865 4153 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:33:54.286178 4168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:54.287313 4153 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:33:54.287420 4153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/ts-1/wal
uuid: "0118eeda11094ec9bedcfb87a9fd5df1"
format_stamp: "Formatted at 2026-09-27 02:33:54 on dist-test-slave-grlk"
server_key: "e4289e52c67443b64796ffc7b4f847e5"
server_key_iv: "86cfc166b7acc4473ad9d065c32639e0"
server_key_version: "encryptionkey@0"
I20260927 02:33:54.287508 4153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:33:54.318073 4153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:33:54.320602 4153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:33:54.320761 4153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:33:54.321295 4153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:33:54.322018 4153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:33:54.322062 4153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:54.322113 4153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:33:54.322144 4153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:54.330076 4153 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36705
I20260927 02:33:54.330127 4281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36705 every 8 connection(s)
I20260927 02:33:54.330924 4153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:33:54.334581 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 4153
I20260927 02:33:54.334682 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:33:54.334910 392 external_mini_cluster.cc:1428] Setting key ce02b478ec5e699c6dbcd5ed9ed26dcf
Sep 27 02:33:54 dist-test-slave-grlk krb5kdc[3912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:33:54.344373 3996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:54.332583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44285 (local address 127.0.98.62:33543)
0927 02:33:54.332693 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:54.332697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:54.333546 (+ 849us) server_negotiation.cc:407] Connection header received
0927 02:33:54.334892 (+ 1346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:54.334895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:54.334950 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:54.335040 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:54.336346 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:54.336956 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:54.338384 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:54.338704 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:54.338832 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:54.341293 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:54.341296 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:54.341298 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:54.341319 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:54.342577 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:54.343043 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:54.343046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:54.343047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:54.343118 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:54.343409 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:54.343413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:54.343415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:54.343578 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:54.343659 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:54.344105 (+ 446us) server_negotiation.cc:299] Negotiation successful
0927 02:33:54.344219 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 02:33:54.345182 4284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:54.332889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33543 (local address 127.0.98.2:44285)
0927 02:33:54.333395 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:54.333436 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:54.334685 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:54.335239 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:54.335248 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:54.335608 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:33:54.336203 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:54.336222 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:54.337073 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:54.337076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:54.338251 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:54.338261 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:54.338866 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:54.338870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:54.339058 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:54.339617 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:33:54.339635 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:33:54.341147 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:33:54.342679 (+ 1532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:54.342685 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:54.342698 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:54.342927 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:54.343198 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:54.343201 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:54.343203 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:54.343325 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:54.343670 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:33:54.343676 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:33:54.343897 (+ 221us) client_negotiation.cc:769] Sending connection context
0927 02:33:54.344053 (+ 156us) client_negotiation.cc:240] Negotiation successful
0927 02:33:54.344340 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":104,"threads_started":1}
I20260927 02:33:54.346196 4282 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33543
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 02:33:54.346424 4282 heartbeater.cc:464] Registering TS with master...
I20260927 02:33:54.346863 4282 heartbeater.cc:511] Master 127.0.98.62:33543 requested a full tablet report, sending...
I20260927 02:33:54.347770 3957 ts_manager.cc:194] Registered new tserver with Master: 0118eeda11094ec9bedcfb87a9fd5df1 (127.0.98.2:36705)
I20260927 02:33:54.348301 3957 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44285
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:33:54.359867 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33543
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:33:54.465632 4287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:33:54.465868 4287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:33:54.465914 4287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:33:54.470217 4287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:33:54.470271 4287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:33:54.470329 4287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:33:54.475458 4287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:33:54.476408 4287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:33:54.477370 4287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:33:54.486048 4293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:54.486076 4295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:54.486188 4287 server_base.cc:1034] running on GCE node
W20260927 02:33:54.486409 4292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:54.486759 4287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:33:54.487321 4287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:33:54.488502 4287 hybrid_clock.cc:648] HybridClock initialized: now 1790476434488475 us; error 28 us; skew 500 ppm
Sep 27 02:33:54 dist-test-slave-grlk krb5kdc[3912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:33:54.491019 4287 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:33:54.491846 4287 webserver.cc:460] Webserver started at http://127.0.98.3:45927/ using document root <none> and password file <none>
I20260927 02:33:54.492321 4287 fs_manager.cc:359] Metadata directory not provided
I20260927 02:33:54.492372 4287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:33:54.492543 4287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:33:54.494479 4287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/ts-2/data/instance:
uuid: "6a2d5e5c2783480da61e03938a0f27a0"
format_stamp: "Formatted at 2026-09-27 02:33:54 on dist-test-slave-grlk"
server_key: "71f61875112a99b824af6468afd5817a"
server_key_iv: "f7816924605f0e92142f837cebcf7746"
server_key_version: "encryptionkey@0"
I20260927 02:33:54.494993 4287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6a2d5e5c2783480da61e03938a0f27a0"
format_stamp: "Formatted at 2026-09-27 02:33:54 on dist-test-slave-grlk"
server_key: "71f61875112a99b824af6468afd5817a"
server_key_iv: "f7816924605f0e92142f837cebcf7746"
server_key_version: "encryptionkey@0"
I20260927 02:33:54.498339 4287 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:33:54.500465 4302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:54.501580 4287 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:33:54.501674 4287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/ts-2/wal
uuid: "6a2d5e5c2783480da61e03938a0f27a0"
format_stamp: "Formatted at 2026-09-27 02:33:54 on dist-test-slave-grlk"
server_key: "71f61875112a99b824af6468afd5817a"
server_key_iv: "f7816924605f0e92142f837cebcf7746"
server_key_version: "encryptionkey@0"
I20260927 02:33:54.501762 4287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:33:54.526810 4287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:33:54.529893 4287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:33:54.530071 4287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:33:54.530617 4287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:33:54.531406 4287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:33:54.531454 4287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:54.531507 4287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:33:54.531530 4287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:54.539844 4287 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44435
I20260927 02:33:54.539886 4415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44435 every 8 connection(s)
I20260927 02:33:54.540675 4287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:33:54.545312 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 4287
I20260927 02:33:54.545418 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476410933508-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:33:54.545661 392 external_mini_cluster.cc:1428] Setting key 5bdc325f3b00b3920e854e4285ffab50
Sep 27 02:33:54 dist-test-slave-grlk krb5kdc[3912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:33:54.553507 3996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:54.542435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47145 (local address 127.0.98.62:33543)
0927 02:33:54.542574 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:54.542579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:54.543381 (+ 802us) server_negotiation.cc:407] Connection header received
0927 02:33:54.544648 (+ 1267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:54.544651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:54.544697 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:54.544767 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:54.545990 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:54.546624 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:54.548048 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:54.548264 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:54.548340 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:54.550593 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:54.550596 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:54.550599 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:54.550624 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:54.551795 (+ 1171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:54.552237 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:54.552241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:54.552242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:54.552321 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:54.552595 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:54.552598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:54.552599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:54.552775 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:54.552850 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:54.553288 (+ 438us) server_negotiation.cc:299] Negotiation successful
0927 02:33:54.553384 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 02:33:54.554296 4418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:54.542803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33543 (local address 127.0.98.3:47145)
0927 02:33:54.543238 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:54.543279 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:54.544454 (+ 1175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:54.544896 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:54.544903 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:54.545327 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:33:54.545853 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:54.545867 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:54.546771 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:54.546775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:54.547943 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:54.547953 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:54.548373 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:54.548376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:54.548506 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:54.549080 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:33:54.549095 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:33:54.550446 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:33:54.551890 (+ 1444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:54.551895 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:54.551906 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:54.552131 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:54.552399 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:54.552401 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:54.552403 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:54.552518 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:54.552849 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:33:54.552853 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:33:54.553098 (+ 245us) client_negotiation.cc:769] Sending connection context
0927 02:33:54.553250 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 02:33:54.553482 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":107,"threads_started":1}
I20260927 02:33:54.555325 4416 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33543
I20260927 02:33:54.555526 4416 heartbeater.cc:464] Registering TS with master...
I20260927 02:33:54.556005 4416 heartbeater.cc:511] Master 127.0.98.62:33543 requested a full tablet report, sending...
I20260927 02:33:54.556913 3957 ts_manager.cc:194] Registered new tserver with Master: 6a2d5e5c2783480da61e03938a0f27a0 (127.0.98.3:44435)
I20260927 02:33:54.557451 3957 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47145
I20260927 02:33:54.559221 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 02:33:54 dist-test-slave-grlk krb5kdc[3912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 02:33:54 dist-test-slave-grlk krb5kdc[3912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:33:54.582053 4423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:54.573874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45870 (local address 127.0.98.1:35241)
0927 02:33:54.574163 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:54.574176 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:54.574201 (+ 25us) server_negotiation.cc:407] Connection header received
0927 02:33:54.574249 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:54.574254 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:54.574443 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:54.574556 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:54.575150 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:54.575905 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:54.576652 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:54.577070 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:54.577150 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:54.579254 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:54.579262 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:54.579274 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:54.579297 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:54.580596 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:54.580997 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:54.581001 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:54.581002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:54.581074 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:54.581289 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:54.581291 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:54.581292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:54.581500 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:54.581682 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:54.581787 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 02:33:54.581908 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":96,"threads_started":1}
Sep 27 02:33:54 dist-test-slave-grlk krb5kdc[3912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:33:54.591826 3996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:54.584736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37630 (local address 127.0.98.62:33543)
0927 02:33:54.584855 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:54.584859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:54.584886 (+ 27us) server_negotiation.cc:407] Connection header received
0927 02:33:54.584987 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:54.584990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:54.585034 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:54.585093 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:54.585630 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:54.586135 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:54.586855 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:54.587071 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:54.587149 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:54.589200 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:54.589203 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:54.589205 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:54.589229 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:54.590580 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:54.590895 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:54.590898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:54.590899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:54.590962 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:54.591135 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:54.591137 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:54.591138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:54.591361 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:54.591489 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:54.591627 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 02:33:54.591711 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 02:33:54.592163 3957 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37630
I20260927 02:33:54.592655 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 4019
I20260927 02:33:54.597445 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 4153
I20260927 02:33:54.601908 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 4287
I20260927 02:33:54.606225 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 3925
2026-09-27T02:33:54Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2009 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:33:54 dist-test-slave-grlk krb5kdc[4428](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:33:54 dist-test-slave-grlk krb5kdc[4428](info): set up 2 sockets
Sep 27 02:33:54 dist-test-slave-grlk krb5kdc[4428](info): commencing operation
krb5kdc: starting...
W20260927 02:33:55.552608 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/krb5kdc/test-user.keytab.
Sep 27 02:33:55 dist-test-slave-grlk krb5kdc[4428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:33:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:33:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:33:55.663509 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33757
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33757
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:33:55.766366 4441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:33:55.766624 4441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:33:55.766680 4441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:33:55.770639 4441 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:33:55.770687 4441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:33:55.770711 4441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:33:55.770732 4441 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:33:55.770761 4441 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:33:55.775838 4441 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33757
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:33:55.776808 4441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:33:55.777774 4441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:33:55.785431 4441 server_base.cc:1034] running on GCE node
W20260927 02:33:55.785378 4446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:55.785547 4447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:55.785382 4449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:55.786167 4441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:33:55.787215 4441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:33:55.788441 4441 hybrid_clock.cc:648] HybridClock initialized: now 1790476435788422 us; error 28 us; skew 500 ppm
Sep 27 02:33:55 dist-test-slave-grlk krb5kdc[4428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476435, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:33:55.791236 4441 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:33:55.791922 4441 fs_manager.cc:359] Metadata directory not provided
I20260927 02:33:55.791996 4441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:33:55.792176 4441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:33:55.794188 4441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "abe60d43d20c467cbdd1fc7cecc87993"
format_stamp: "Formatted at 2026-09-27 02:33:55 on dist-test-slave-grlk"
server_key: "cc525d8fc248a70d530b7b7a863e273c"
server_key_iv: "8febd2bf6b9a6f83f59a4b0a524abb08"
server_key_version: "encryptionkey@0"
I20260927 02:33:55.794719 4441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "abe60d43d20c467cbdd1fc7cecc87993"
format_stamp: "Formatted at 2026-09-27 02:33:55 on dist-test-slave-grlk"
server_key: "cc525d8fc248a70d530b7b7a863e273c"
server_key_iv: "8febd2bf6b9a6f83f59a4b0a524abb08"
server_key_version: "encryptionkey@0"
I20260927 02:33:55.798295 4441 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:33:55.800491 4455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:55.801673 4441 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:33:55.801789 4441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "abe60d43d20c467cbdd1fc7cecc87993"
format_stamp: "Formatted at 2026-09-27 02:33:55 on dist-test-slave-grlk"
server_key: "cc525d8fc248a70d530b7b7a863e273c"
server_key_iv: "8febd2bf6b9a6f83f59a4b0a524abb08"
server_key_version: "encryptionkey@0"
I20260927 02:33:55.801882 4441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:33:55.812103 4441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:33:55.814597 4441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:33:55.814750 4441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:33:55.821152 4441 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33757
I20260927 02:33:55.821401 4507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33757 every 8 connection(s)
I20260927 02:33:55.821808 4441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:33:55.824349 4508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:33:55.828433 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 4441
I20260927 02:33:55.828527 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:33:55.828770 392 external_mini_cluster.cc:1428] Setting key e67877a5e8628d2779215150ac140d16
I20260927 02:33:55.829679 4508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abe60d43d20c467cbdd1fc7cecc87993: Bootstrap starting.
I20260927 02:33:55.832208 4508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abe60d43d20c467cbdd1fc7cecc87993: Neither blocks nor log segments found. Creating new log.
I20260927 02:33:55.833021 4508 log.cc:824] T 00000000000000000000000000000000 P abe60d43d20c467cbdd1fc7cecc87993: Log is configured to *not* fsync() on all Append() calls
I20260927 02:33:55.834861 4508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abe60d43d20c467cbdd1fc7cecc87993: No bootstrap required, opened a new log
Sep 27 02:33:55 dist-test-slave-grlk krb5kdc[4428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:33:55.838558 4508 raft_consensus.cc:348] T 00000000000000000000000000000000 P abe60d43d20c467cbdd1fc7cecc87993 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe60d43d20c467cbdd1fc7cecc87993" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33757 } }
I20260927 02:33:55.838817 4508 raft_consensus.cc:374] T 00000000000000000000000000000000 P abe60d43d20c467cbdd1fc7cecc87993 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:33:55.838873 4508 raft_consensus.cc:729] T 00000000000000000000000000000000 P abe60d43d20c467cbdd1fc7cecc87993 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abe60d43d20c467cbdd1fc7cecc87993, State: Initialized, Role: FOLLOWER
I20260927 02:33:55.839408 4508 consensus_queue.cc:260] T 00000000000000000000000000000000 P abe60d43d20c467cbdd1fc7cecc87993 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe60d43d20c467cbdd1fc7cecc87993" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33757 } }
I20260927 02:33:55.839560 4508 raft_consensus.cc:388] T 00000000000000000000000000000000 P abe60d43d20c467cbdd1fc7cecc87993 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:33:55.839620 4508 raft_consensus.cc:482] T 00000000000000000000000000000000 P abe60d43d20c467cbdd1fc7cecc87993 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:33:55.839707 4508 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abe60d43d20c467cbdd1fc7cecc87993 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:33:55.840631 4508 raft_consensus.cc:504] T 00000000000000000000000000000000 P abe60d43d20c467cbdd1fc7cecc87993 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe60d43d20c467cbdd1fc7cecc87993" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33757 } }
I20260927 02:33:55.841003 4508 leader_election.cc:302] T 00000000000000000000000000000000 P abe60d43d20c467cbdd1fc7cecc87993 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abe60d43d20c467cbdd1fc7cecc87993; no voters:
I20260927 02:33:55.841322 4508 leader_election.cc:288] T 00000000000000000000000000000000 P abe60d43d20c467cbdd1fc7cecc87993 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:33:55.841403 4513 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abe60d43d20c467cbdd1fc7cecc87993 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:33:55.841770 4513 raft_consensus.cc:686] T 00000000000000000000000000000000 P abe60d43d20c467cbdd1fc7cecc87993 [term 1 LEADER]: Becoming Leader. State: Replica: abe60d43d20c467cbdd1fc7cecc87993, State: Running, Role: LEADER
I20260927 02:33:55.842193 4513 consensus_queue.cc:237] T 00000000000000000000000000000000 P abe60d43d20c467cbdd1fc7cecc87993 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe60d43d20c467cbdd1fc7cecc87993" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33757 } }
I20260927 02:33:55.842540 4508 sys_catalog.cc:564] T 00000000000000000000000000000000 P abe60d43d20c467cbdd1fc7cecc87993 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:33:55.843170 4511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:55.830113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47392 (local address 127.0.98.62:33757)
0927 02:33:55.830510 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:55.830526 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:55.830566 (+ 40us) server_negotiation.cc:407] Connection header received
0927 02:33:55.831360 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:55.831383 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:55.831790 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:55.832126 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:55.832762 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:55.834104 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:55.834960 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:55.835607 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:55.835756 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:55.838140 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:55.838146 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:55.838161 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:55.838189 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:55.839986 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:55.840397 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:55.840403 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:55.840409 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:55.840555 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:55.840807 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:55.840811 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:55.840812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:55.841318 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:55.841525 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:55.841887 (+ 362us) server_negotiation.cc:299] Negotiation successful
0927 02:33:55.842163 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":102,"threads_started":1}
I20260927 02:33:55.843381 4513 sys_catalog.cc:455] T 00000000000000000000000000000000 P abe60d43d20c467cbdd1fc7cecc87993 [sys.catalog]: SysCatalogTable state changed. Reason: New leader abe60d43d20c467cbdd1fc7cecc87993. Latest consensus state: current_term: 1 leader_uuid: "abe60d43d20c467cbdd1fc7cecc87993" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe60d43d20c467cbdd1fc7cecc87993" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33757 } } }
I20260927 02:33:55.843485 4513 sys_catalog.cc:458] T 00000000000000000000000000000000 P abe60d43d20c467cbdd1fc7cecc87993 [sys.catalog]: This master's current role is: LEADER
I20260927 02:33:55.844092 4514 sys_catalog.cc:455] T 00000000000000000000000000000000 P abe60d43d20c467cbdd1fc7cecc87993 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abe60d43d20c467cbdd1fc7cecc87993" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe60d43d20c467cbdd1fc7cecc87993" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33757 } } }
I20260927 02:33:55.844259 4514 sys_catalog.cc:458] T 00000000000000000000000000000000 P abe60d43d20c467cbdd1fc7cecc87993 [sys.catalog]: This master's current role is: LEADER
W20260927 02:33:55.847499 4528 catalog_manager.cc:1558] T 00000000000000000000000000000000 P abe60d43d20c467cbdd1fc7cecc87993: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:33:55.847635 4528 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:33:55.847890 4522 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:33:55.848714 4522 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:33:55.854482 4522 catalog_manager.cc:1347] Generated new cluster ID: f924ccdf8f2143f282f8383cf94a056d
I20260927 02:33:55.854550 4522 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:33:55.874116 4522 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:33:55.875339 4522 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:33:55.880366 4522 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abe60d43d20c467cbdd1fc7cecc87993: Generated new TSK 0
I20260927 02:33:55.881168 4522 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:33:55.914100 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33757
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:33:56.021728 4534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:33:56.021973 4534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:33:56.022017 4534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:33:56.026052 4534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:33:56.026104 4534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:33:56.026171 4534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:33:56.031409 4534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:33:56.032387 4534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:33:56.033397 4534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:33:56.042791 4542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:56.042793 4540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:56.042793 4539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:56.042949 4534 server_base.cc:1034] running on GCE node
I20260927 02:33:56.043594 4534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:33:56.044271 4534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:33:56.045444 4534 hybrid_clock.cc:648] HybridClock initialized: now 1790476436045409 us; error 47 us; skew 500 ppm
Sep 27 02:33:56 dist-test-slave-grlk krb5kdc[4428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:33:56.048275 4534 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:33:56.049010 4534 fs_manager.cc:359] Metadata directory not provided
I20260927 02:33:56.049077 4534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:33:56.049244 4534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:33:56.051172 4534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-0/data/instance:
uuid: "7b3985cfef1a481b970702e57b715f8c"
format_stamp: "Formatted at 2026-09-27 02:33:56 on dist-test-slave-grlk"
server_key: "4534359332f9cbe63cb341e315f3a6cf"
server_key_iv: "5448919aaf31c3a97012c6677459ae23"
server_key_version: "encryptionkey@0"
I20260927 02:33:56.051649 4534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7b3985cfef1a481b970702e57b715f8c"
format_stamp: "Formatted at 2026-09-27 02:33:56 on dist-test-slave-grlk"
server_key: "4534359332f9cbe63cb341e315f3a6cf"
server_key_iv: "5448919aaf31c3a97012c6677459ae23"
server_key_version: "encryptionkey@0"
I20260927 02:33:56.055163 4534 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:33:56.057488 4548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:56.058629 4534 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:33:56.058723 4534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-0/wal
uuid: "7b3985cfef1a481b970702e57b715f8c"
format_stamp: "Formatted at 2026-09-27 02:33:56 on dist-test-slave-grlk"
server_key: "4534359332f9cbe63cb341e315f3a6cf"
server_key_iv: "5448919aaf31c3a97012c6677459ae23"
server_key_version: "encryptionkey@0"
I20260927 02:33:56.058804 4534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:33:56.082206 4534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:33:56.085206 4534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:33:56.085364 4534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:33:56.085873 4534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:33:56.086645 4534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:33:56.086691 4534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:56.086730 4534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:33:56.086750 4534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:56.094835 4534 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44915
I20260927 02:33:56.094861 4661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44915 every 8 connection(s)
I20260927 02:33:56.095398 4534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:33:56.100556 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 4534
I20260927 02:33:56.100675 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:33:56.101030 392 external_mini_cluster.cc:1428] Setting key 6f1e1fb918d3e1cc16996bc93fd98ce5
Sep 27 02:33:56 dist-test-slave-grlk krb5kdc[4428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:33:56.108769 4511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:56.097123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37805 (local address 127.0.98.62:33757)
0927 02:33:56.097265 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:56.097270 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:56.098127 (+ 857us) server_negotiation.cc:407] Connection header received
0927 02:33:56.099378 (+ 1251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:56.099381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:56.099433 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:56.099505 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:56.100591 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:56.101335 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:56.102609 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:56.102833 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:56.102915 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:56.105530 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:56.105533 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:56.105535 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:56.105556 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:56.106912 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:56.107356 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:56.107360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:56.107362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:56.107433 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:56.107722 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:56.107725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:56.107727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:56.107886 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:56.107975 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:56.108488 (+ 513us) server_negotiation.cc:299] Negotiation successful
0927 02:33:56.108614 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:33:56.109562 4664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:56.097568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33757 (local address 127.0.98.1:37805)
0927 02:33:56.097987 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:56.098029 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:56.099201 (+ 1172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:56.099652 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:56.099658 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:56.099987 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:33:56.100463 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:56.100477 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:56.101446 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:56.101450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:56.102489 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:56.102507 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:56.102977 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:56.102982 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:56.103129 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:56.103675 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:33:56.103694 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:33:56.105367 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:33:56.107023 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:56.107028 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:56.107041 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:56.107268 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:56.107511 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:56.107514 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:56.107516 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:56.107641 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:56.107978 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:33:56.107984 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:33:56.108274 (+ 290us) client_negotiation.cc:769] Sending connection context
0927 02:33:56.108453 (+ 179us) client_negotiation.cc:240] Negotiation successful
0927 02:33:56.108743 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":107,"threads_started":1}
I20260927 02:33:56.110623 4662 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33757
I20260927 02:33:56.110849 4662 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 02:33:56.111311 4662 heartbeater.cc:511] Master 127.0.98.62:33757 requested a full tablet report, sending...
I20260927 02:33:56.112748 4472 ts_manager.cc:194] Registered new tserver with Master: 7b3985cfef1a481b970702e57b715f8c (127.0.98.1:44915)
I20260927 02:33:56.113905 4472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37805
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:33:56.123767 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33757
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:33:56.227891 4667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:33:56.228127 4667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:33:56.228168 4667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:33:56.232097 4667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:33:56.232149 4667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:33:56.232206 4667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:33:56.237247 4667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:33:56.238121 4667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:33:56.239148 4667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:33:56.247792 4675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:56.247792 4672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:56.247808 4673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:56.248153 4667 server_base.cc:1034] running on GCE node
I20260927 02:33:56.248564 4667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:33:56.249187 4667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:33:56.250330 4667 hybrid_clock.cc:648] HybridClock initialized: now 1790476436250313 us; error 32 us; skew 500 ppm
Sep 27 02:33:56 dist-test-slave-grlk krb5kdc[4428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:33:56.252733 4667 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:33:56.253405 4667 fs_manager.cc:359] Metadata directory not provided
I20260927 02:33:56.253465 4667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:33:56.253643 4667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:33:56.255501 4667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-1/data/instance:
uuid: "8ef2f13ba55a4b27b870d6d8fcaf81d8"
format_stamp: "Formatted at 2026-09-27 02:33:56 on dist-test-slave-grlk"
server_key: "808e78c0cb32b87ab5f9435f8ebcd38d"
server_key_iv: "bda4f28e2a8711da88c387c6c3b0cf8e"
server_key_version: "encryptionkey@0"
I20260927 02:33:56.256009 4667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8ef2f13ba55a4b27b870d6d8fcaf81d8"
format_stamp: "Formatted at 2026-09-27 02:33:56 on dist-test-slave-grlk"
server_key: "808e78c0cb32b87ab5f9435f8ebcd38d"
server_key_iv: "bda4f28e2a8711da88c387c6c3b0cf8e"
server_key_version: "encryptionkey@0"
I20260927 02:33:56.259317 4667 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:33:56.261423 4681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:56.262552 4667 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:33:56.262660 4667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-1/wal
uuid: "8ef2f13ba55a4b27b870d6d8fcaf81d8"
format_stamp: "Formatted at 2026-09-27 02:33:56 on dist-test-slave-grlk"
server_key: "808e78c0cb32b87ab5f9435f8ebcd38d"
server_key_iv: "bda4f28e2a8711da88c387c6c3b0cf8e"
server_key_version: "encryptionkey@0"
I20260927 02:33:56.262738 4667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:33:56.280735 4667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:33:56.283386 4667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:33:56.283545 4667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:33:56.284063 4667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:33:56.284818 4667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:33:56.284863 4667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:56.284914 4667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:33:56.284960 4667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:56.294165 4667 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33357
I20260927 02:33:56.294181 4794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33357 every 8 connection(s)
I20260927 02:33:56.294734 4667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:33:56.299536 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 4667
I20260927 02:33:56.299659 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:33:56.300081 392 external_mini_cluster.cc:1428] Setting key aaa452eae11892509fd36975a496f9a7
Sep 27 02:33:56 dist-test-slave-grlk krb5kdc[4428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:33:56.308176 4511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:56.296411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39591 (local address 127.0.98.62:33757)
0927 02:33:56.296561 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:56.296564 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:56.297382 (+ 818us) server_negotiation.cc:407] Connection header received
0927 02:33:56.298665 (+ 1283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:56.298668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:56.298723 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:56.298795 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:56.299922 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:56.300462 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:56.301616 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:56.301829 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:56.301914 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:56.304348 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:56.304352 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:56.304355 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:56.304381 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:56.306125 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:56.306649 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:56.306652 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:56.306654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:56.306741 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:56.307019 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:56.307022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:56.307023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:56.307238 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:56.307337 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:56.307757 (+ 420us) server_negotiation.cc:299] Negotiation successful
0927 02:33:56.307869 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:33:56.308704 4797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:56.296778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33757 (local address 127.0.98.2:39591)
0927 02:33:56.297247 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:56.297287 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:56.298471 (+ 1184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:56.298964 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:56.298971 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:56.299307 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:33:56.299794 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:56.299808 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:56.300577 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:56.300580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:56.301496 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:56.301505 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:56.301960 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:56.301964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:56.302133 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:56.302711 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:33:56.302734 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:33:56.304184 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:33:56.306269 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:56.306276 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:56.306290 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:56.306551 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:56.306824 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:56.306826 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:56.306827 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:56.306942 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:56.307324 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:33:56.307328 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:33:56.307532 (+ 204us) client_negotiation.cc:769] Sending connection context
0927 02:33:56.307684 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 02:33:56.307906 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":145,"threads_started":1}
I20260927 02:33:56.309760 4795 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33757
I20260927 02:33:56.309962 4795 heartbeater.cc:464] Registering TS with master...
I20260927 02:33:56.310391 4795 heartbeater.cc:511] Master 127.0.98.62:33757 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 02:33:56.311362 4472 ts_manager.cc:194] Registered new tserver with Master: 8ef2f13ba55a4b27b870d6d8fcaf81d8 (127.0.98.2:33357)
I20260927 02:33:56.312033 4472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39591
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:33:56.324638 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33757
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:33:56.429380 4800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:33:56.429602 4800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:33:56.429651 4800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:33:56.433533 4800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:33:56.433588 4800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:33:56.433655 4800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:33:56.438679 4800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:33:56.439728 4800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:33:56.440815 4800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:33:56.449560 4808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:56.449564 4806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:56.449690 4800 server_base.cc:1034] running on GCE node
W20260927 02:33:56.449836 4805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:56.450187 4800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:33:56.450732 4800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:33:56.451872 4800 hybrid_clock.cc:648] HybridClock initialized: now 1790476436451856 us; error 28 us; skew 500 ppm
Sep 27 02:33:56 dist-test-slave-grlk krb5kdc[4428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:33:56.454396 4800 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:33:56.455017 4800 fs_manager.cc:359] Metadata directory not provided
I20260927 02:33:56.455082 4800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:33:56.455250 4800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:33:56.457139 4800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-2/data/instance:
uuid: "f6af42cd10de4e1fa09dee599b93cc8f"
format_stamp: "Formatted at 2026-09-27 02:33:56 on dist-test-slave-grlk"
server_key: "9fa3988e23c923947dac318acb736638"
server_key_iv: "1e7c175e827209ec4a507c8c3e8ef4ec"
server_key_version: "encryptionkey@0"
I20260927 02:33:56.457628 4800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f6af42cd10de4e1fa09dee599b93cc8f"
format_stamp: "Formatted at 2026-09-27 02:33:56 on dist-test-slave-grlk"
server_key: "9fa3988e23c923947dac318acb736638"
server_key_iv: "1e7c175e827209ec4a507c8c3e8ef4ec"
server_key_version: "encryptionkey@0"
I20260927 02:33:56.460860 4800 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:33:56.463054 4814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:56.464134 4800 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 02:33:56.464232 4800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-2/wal
uuid: "f6af42cd10de4e1fa09dee599b93cc8f"
format_stamp: "Formatted at 2026-09-27 02:33:56 on dist-test-slave-grlk"
server_key: "9fa3988e23c923947dac318acb736638"
server_key_iv: "1e7c175e827209ec4a507c8c3e8ef4ec"
server_key_version: "encryptionkey@0"
I20260927 02:33:56.464332 4800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:33:56.477478 4800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:33:56.479866 4800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:33:56.480023 4800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:33:56.480535 4800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:33:56.481340 4800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:33:56.481387 4800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:56.481436 4800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:33:56.481463 4800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:56.489280 4800 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44695
I20260927 02:33:56.489301 4927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44695 every 8 connection(s)
I20260927 02:33:56.489810 4800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 02:33:56 dist-test-slave-grlk krb5kdc[4428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:33:56.499713 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 4800
I20260927 02:33:56.499800 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:33:56.500108 392 external_mini_cluster.cc:1428] Setting key b589b2a409e309be57861ba0e1594c12
I20260927 02:33:56.501915 4511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:56.491345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37297 (local address 127.0.98.62:33757)
0927 02:33:56.491482 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:56.491487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:56.492337 (+ 850us) server_negotiation.cc:407] Connection header received
0927 02:33:56.493619 (+ 1282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:56.493621 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:56.493678 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:56.493749 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:56.494795 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:56.495311 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:56.496365 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:56.496569 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:56.496634 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:56.498736 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:56.498739 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:56.498740 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:56.498764 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:56.499896 (+ 1132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:56.500394 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:56.500397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:56.500399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:56.500488 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:56.500811 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:56.500814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:56.500816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:56.501074 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:56.501151 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:56.501659 (+ 508us) server_negotiation.cc:299] Negotiation successful
0927 02:33:56.501771 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 02:33:56.502640 4930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:56.491736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33757 (local address 127.0.98.3:37297)
0927 02:33:56.492188 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:56.492228 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:56.493416 (+ 1188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:56.493904 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:56.493911 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:56.494239 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:33:56.494691 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:56.494703 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:56.495412 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:56.495415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:56.496279 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:56.496287 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:56.496665 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:56.496667 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:56.496791 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:56.497294 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:33:56.497308 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:33:56.498597 (+ 1289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:33:56.499994 (+ 1397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:56.499999 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:56.500012 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:56.500311 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:56.500566 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:33:56.500568 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:33:56.500570 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:33:56.500729 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:33:56.501170 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:33:56.501176 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:33:56.501441 (+ 265us) client_negotiation.cc:769] Sending connection context
0927 02:33:56.501595 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 02:33:56.501819 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":111,"threads_started":1}
I20260927 02:33:56.503566 4928 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33757
I20260927 02:33:56.503767 4928 heartbeater.cc:464] Registering TS with master...
I20260927 02:33:56.504173 4928 heartbeater.cc:511] Master 127.0.98.62:33757 requested a full tablet report, sending...
I20260927 02:33:56.505080 4472 ts_manager.cc:194] Registered new tserver with Master: f6af42cd10de4e1fa09dee599b93cc8f (127.0.98.3:44695)
I20260927 02:33:56.505585 4472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37297
I20260927 02:33:56.513370 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:33:56.521629 4511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:56.515047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47404 (local address 127.0.98.62:33757)
0927 02:33:56.515152 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:56.515157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:56.515302 (+ 145us) server_negotiation.cc:407] Connection header received
0927 02:33:56.515420 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:56.515423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:56.515463 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:56.515519 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:56.516132 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:56.516623 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:56.517427 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:56.517684 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:56.517765 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:56.518906 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:56.518910 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:56.518913 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:33:56.518936 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:56.520129 (+ 1193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:56.520490 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:56.520493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:56.520495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:56.520571 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:33:56.520800 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:33:56.520803 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:33:56.520804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:33:56.521067 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:56.521181 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:56.521387 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 02:33:56.521486 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 02:33:56.524163 4472 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47404:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:33:56.527081 4472 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:33:56.540251 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:56.534671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44695 (local address 127.0.0.1:50652)
0927 02:33:56.535114 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:56.535126 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:56.535191 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:56.536364 (+ 1173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:56.536367 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:56.536385 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:33:56.536529 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:56.536535 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:56.537819 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:56.537823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:56.538848 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:56.538857 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:56.539754 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:56.539758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:56.539864 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:56.539908 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 02:33:56.539998 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 02:33:56.540107 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"mutex_wait_us":61,"spinlock_wait_cycles":1152,"thread_start_us":80,"threads_started":1}
I20260927 02:33:56.540388 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:56.535402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39682 (local address 127.0.98.2:33357)
0927 02:33:56.535781 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:56.535791 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:56.535811 (+ 20us) server_negotiation.cc:407] Connection header received
0927 02:33:56.535881 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:56.535887 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:56.536257 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:56.536658 (+ 401us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:33:56.536777 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:56.537538 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:56.539273 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:56.539811 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:56.539940 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:56.540081 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:56.540166 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 02:33:56.540254 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":114,"threads_started":1}
I20260927 02:33:56.540596 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:56.535215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33357 (local address 127.0.0.1:39682)
0927 02:33:56.535471 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:56.535483 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:56.535580 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:56.536391 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:56.536394 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:56.536416 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:33:56.536520 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:56.536527 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:56.537660 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:56.537664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:56.539136 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:56.539146 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:56.539967 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:56.539970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:56.540051 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:56.540075 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 02:33:56.540141 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 02:33:56.540208 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":73,"spinlock_wait_cycles":4608,"thread_start_us":84,"threads_started":1}
I20260927 02:33:56.540601 4940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:56.534543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44915 (local address 127.0.0.1:42570)
0927 02:33:56.535023 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:56.535071 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:56.535191 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:56.536109 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:56.536114 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:56.536135 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:33:56.536252 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:56.536259 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:56.537684 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:56.537687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:56.539136 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:56.539146 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:56.540264 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:56.540267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:56.540351 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:56.540372 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 02:33:56.540444 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 02:33:56.540501 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":159,"thread_start_us":71,"threads_started":1}
I20260927 02:33:56.540887 4941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:56.534718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42570 (local address 127.0.98.1:44915)
0927 02:33:56.535455 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:56.535467 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:56.535487 (+ 20us) server_negotiation.cc:407] Connection header received
0927 02:33:56.535559 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:56.535564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:56.535764 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:56.535916 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:33:56.536370 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:56.537538 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:56.539259 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:56.540000 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:56.540091 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:56.540211 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:56.540663 (+ 452us) server_negotiation.cc:299] Negotiation successful
0927 02:33:56.540753 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":139,"threads_started":1}
I20260927 02:33:56.540920 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:56.535528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50652 (local address 127.0.98.3:44695)
0927 02:33:56.535923 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:56.535936 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:56.535955 (+ 19us) server_negotiation.cc:407] Connection header received
0927 02:33:56.536023 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:56.536027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:56.536203 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:56.536381 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:33:56.536629 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:56.537707 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:56.538980 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:56.539635 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:56.539740 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:56.539881 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:56.540698 (+ 817us) server_negotiation.cc:299] Negotiation successful
0927 02:33:56.540779 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":124,"threads_started":1}
I20260927 02:33:56.543242 4596 tablet_service.cc:1467] Processing CreateTablet for tablet d9f4644ceb764927a4f070b3c74f49a7 (DEFAULT_TABLE table=test-table [id=6eb40e0eafae40df87efcdf696c298d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:33:56.543243 4729 tablet_service.cc:1467] Processing CreateTablet for tablet d9f4644ceb764927a4f070b3c74f49a7 (DEFAULT_TABLE table=test-table [id=6eb40e0eafae40df87efcdf696c298d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:33:56.544137 4729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9f4644ceb764927a4f070b3c74f49a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:33:56.544137 4596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9f4644ceb764927a4f070b3c74f49a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:33:56.546797 4862 tablet_service.cc:1467] Processing CreateTablet for tablet d9f4644ceb764927a4f070b3c74f49a7 (DEFAULT_TABLE table=test-table [id=6eb40e0eafae40df87efcdf696c298d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:33:56.547725 4862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9f4644ceb764927a4f070b3c74f49a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:33:56.549826 4945 tablet_bootstrap.cc:492] T d9f4644ceb764927a4f070b3c74f49a7 P 8ef2f13ba55a4b27b870d6d8fcaf81d8: Bootstrap starting.
I20260927 02:33:56.551081 4946 tablet_bootstrap.cc:492] T d9f4644ceb764927a4f070b3c74f49a7 P 7b3985cfef1a481b970702e57b715f8c: Bootstrap starting.
I20260927 02:33:56.552439 4945 tablet_bootstrap.cc:654] T d9f4644ceb764927a4f070b3c74f49a7 P 8ef2f13ba55a4b27b870d6d8fcaf81d8: Neither blocks nor log segments found. Creating new log.
I20260927 02:33:56.553148 4946 tablet_bootstrap.cc:654] T d9f4644ceb764927a4f070b3c74f49a7 P 7b3985cfef1a481b970702e57b715f8c: Neither blocks nor log segments found. Creating new log.
I20260927 02:33:56.553700 4945 log.cc:824] T d9f4644ceb764927a4f070b3c74f49a7 P 8ef2f13ba55a4b27b870d6d8fcaf81d8: Log is configured to *not* fsync() on all Append() calls
I20260927 02:33:56.553814 4947 tablet_bootstrap.cc:492] T d9f4644ceb764927a4f070b3c74f49a7 P f6af42cd10de4e1fa09dee599b93cc8f: Bootstrap starting.
I20260927 02:33:56.554446 4946 log.cc:824] T d9f4644ceb764927a4f070b3c74f49a7 P 7b3985cfef1a481b970702e57b715f8c: Log is configured to *not* fsync() on all Append() calls
I20260927 02:33:56.555708 4947 tablet_bootstrap.cc:654] T d9f4644ceb764927a4f070b3c74f49a7 P f6af42cd10de4e1fa09dee599b93cc8f: Neither blocks nor log segments found. Creating new log.
I20260927 02:33:56.555727 4945 tablet_bootstrap.cc:492] T d9f4644ceb764927a4f070b3c74f49a7 P 8ef2f13ba55a4b27b870d6d8fcaf81d8: No bootstrap required, opened a new log
I20260927 02:33:56.555895 4945 ts_tablet_manager.cc:1397] T d9f4644ceb764927a4f070b3c74f49a7 P 8ef2f13ba55a4b27b870d6d8fcaf81d8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 02:33:56.555929 4946 tablet_bootstrap.cc:492] T d9f4644ceb764927a4f070b3c74f49a7 P 7b3985cfef1a481b970702e57b715f8c: No bootstrap required, opened a new log
I20260927 02:33:56.556092 4946 ts_tablet_manager.cc:1397] T d9f4644ceb764927a4f070b3c74f49a7 P 7b3985cfef1a481b970702e57b715f8c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 02:33:56.556912 4947 log.cc:824] T d9f4644ceb764927a4f070b3c74f49a7 P f6af42cd10de4e1fa09dee599b93cc8f: Log is configured to *not* fsync() on all Append() calls
I20260927 02:33:56.558764 4947 tablet_bootstrap.cc:492] T d9f4644ceb764927a4f070b3c74f49a7 P f6af42cd10de4e1fa09dee599b93cc8f: No bootstrap required, opened a new log
I20260927 02:33:56.558933 4947 ts_tablet_manager.cc:1397] T d9f4644ceb764927a4f070b3c74f49a7 P f6af42cd10de4e1fa09dee599b93cc8f: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260927 02:33:56.558998 4946 raft_consensus.cc:348] T d9f4644ceb764927a4f070b3c74f49a7 P 7b3985cfef1a481b970702e57b715f8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b3985cfef1a481b970702e57b715f8c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44915 } } peers { permanent_uuid: "f6af42cd10de4e1fa09dee599b93cc8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44695 } } peers { permanent_uuid: "8ef2f13ba55a4b27b870d6d8fcaf81d8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33357 } }
I20260927 02:33:56.559206 4946 raft_consensus.cc:374] T d9f4644ceb764927a4f070b3c74f49a7 P 7b3985cfef1a481b970702e57b715f8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:33:56.559273 4946 raft_consensus.cc:729] T d9f4644ceb764927a4f070b3c74f49a7 P 7b3985cfef1a481b970702e57b715f8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b3985cfef1a481b970702e57b715f8c, State: Initialized, Role: FOLLOWER
I20260927 02:33:56.559676 4946 consensus_queue.cc:260] T d9f4644ceb764927a4f070b3c74f49a7 P 7b3985cfef1a481b970702e57b715f8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b3985cfef1a481b970702e57b715f8c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44915 } } peers { permanent_uuid: "f6af42cd10de4e1fa09dee599b93cc8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44695 } } peers { permanent_uuid: "8ef2f13ba55a4b27b870d6d8fcaf81d8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33357 } }
I20260927 02:33:56.559599 4945 raft_consensus.cc:348] T d9f4644ceb764927a4f070b3c74f49a7 P 8ef2f13ba55a4b27b870d6d8fcaf81d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b3985cfef1a481b970702e57b715f8c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44915 } } peers { permanent_uuid: "f6af42cd10de4e1fa09dee599b93cc8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44695 } } peers { permanent_uuid: "8ef2f13ba55a4b27b870d6d8fcaf81d8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33357 } }
I20260927 02:33:56.559866 4945 raft_consensus.cc:374] T d9f4644ceb764927a4f070b3c74f49a7 P 8ef2f13ba55a4b27b870d6d8fcaf81d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:33:56.559931 4945 raft_consensus.cc:729] T d9f4644ceb764927a4f070b3c74f49a7 P 8ef2f13ba55a4b27b870d6d8fcaf81d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ef2f13ba55a4b27b870d6d8fcaf81d8, State: Initialized, Role: FOLLOWER
I20260927 02:33:56.560323 4945 consensus_queue.cc:260] T d9f4644ceb764927a4f070b3c74f49a7 P 8ef2f13ba55a4b27b870d6d8fcaf81d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b3985cfef1a481b970702e57b715f8c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44915 } } peers { permanent_uuid: "f6af42cd10de4e1fa09dee599b93cc8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44695 } } peers { permanent_uuid: "8ef2f13ba55a4b27b870d6d8fcaf81d8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33357 } }
I20260927 02:33:56.560459 4946 ts_tablet_manager.cc:1428] T d9f4644ceb764927a4f070b3c74f49a7 P 7b3985cfef1a481b970702e57b715f8c: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260927 02:33:56.560806 4662 heartbeater.cc:503] Master 127.0.98.62:33757 was elected leader, sending a full tablet report...
I20260927 02:33:56.560900 4795 heartbeater.cc:503] Master 127.0.98.62:33757 was elected leader, sending a full tablet report...
I20260927 02:33:56.562352 4945 ts_tablet_manager.cc:1428] T d9f4644ceb764927a4f070b3c74f49a7 P 8ef2f13ba55a4b27b870d6d8fcaf81d8: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 02:33:56.563079 4947 raft_consensus.cc:348] T d9f4644ceb764927a4f070b3c74f49a7 P f6af42cd10de4e1fa09dee599b93cc8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b3985cfef1a481b970702e57b715f8c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44915 } } peers { permanent_uuid: "f6af42cd10de4e1fa09dee599b93cc8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44695 } } peers { permanent_uuid: "8ef2f13ba55a4b27b870d6d8fcaf81d8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33357 } }
I20260927 02:33:56.563328 4947 raft_consensus.cc:374] T d9f4644ceb764927a4f070b3c74f49a7 P f6af42cd10de4e1fa09dee599b93cc8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:33:56.563416 4947 raft_consensus.cc:729] T d9f4644ceb764927a4f070b3c74f49a7 P f6af42cd10de4e1fa09dee599b93cc8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6af42cd10de4e1fa09dee599b93cc8f, State: Initialized, Role: FOLLOWER
I20260927 02:33:56.563835 4947 consensus_queue.cc:260] T d9f4644ceb764927a4f070b3c74f49a7 P f6af42cd10de4e1fa09dee599b93cc8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b3985cfef1a481b970702e57b715f8c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44915 } } peers { permanent_uuid: "f6af42cd10de4e1fa09dee599b93cc8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44695 } } peers { permanent_uuid: "8ef2f13ba55a4b27b870d6d8fcaf81d8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33357 } }
I20260927 02:33:56.564535 4928 heartbeater.cc:503] Master 127.0.98.62:33757 was elected leader, sending a full tablet report...
I20260927 02:33:56.564723 4947 ts_tablet_manager.cc:1428] T d9f4644ceb764927a4f070b3c74f49a7 P f6af42cd10de4e1fa09dee599b93cc8f: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
W20260927 02:33:56.596449 4663 tablet.cc:2369] T d9f4644ceb764927a4f070b3c74f49a7 P 7b3985cfef1a481b970702e57b715f8c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:33:56.613802 4952 raft_consensus.cc:482] T d9f4644ceb764927a4f070b3c74f49a7 P 8ef2f13ba55a4b27b870d6d8fcaf81d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:33:56.613998 4952 raft_consensus.cc:504] T d9f4644ceb764927a4f070b3c74f49a7 P 8ef2f13ba55a4b27b870d6d8fcaf81d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b3985cfef1a481b970702e57b715f8c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44915 } } peers { permanent_uuid: "f6af42cd10de4e1fa09dee599b93cc8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44695 } } peers { permanent_uuid: "8ef2f13ba55a4b27b870d6d8fcaf81d8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33357 } }
I20260927 02:33:56.615193 4952 leader_election.cc:288] T d9f4644ceb764927a4f070b3c74f49a7 P 8ef2f13ba55a4b27b870d6d8fcaf81d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b3985cfef1a481b970702e57b715f8c (127.0.98.1:44915), f6af42cd10de4e1fa09dee599b93cc8f (127.0.98.3:44695)
I20260927 02:33:56.620226 4797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:56.615408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44695 (local address 127.0.98.2:57455)
0927 02:33:56.615508 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:56.615524 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:56.615613 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:56.616206 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:56.616209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:56.616228 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:33:56.616374 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:56.616381 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:56.617366 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:56.617369 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:56.618881 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:56.618887 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:56.619701 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:56.619705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:56.619843 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:56.619895 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 02:33:56.619971 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 02:33:56.620048 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":197}
I20260927 02:33:56.620239 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:56.615852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57455 (local address 127.0.98.3:44695)
0927 02:33:56.615977 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:56.615981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:56.615996 (+ 15us) server_negotiation.cc:407] Connection header received
0927 02:33:56.616039 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:56.616042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:56.616088 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:56.616168 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:33:56.616485 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:56.617279 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:56.618991 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:56.619597 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:56.619683 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:56.619743 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:56.620035 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 02:33:56.620111 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 02:33:56.620239 4941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:56.615556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48291 (local address 127.0.98.1:44915)
0927 02:33:56.615670 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:56.615675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:56.615951 (+ 276us) server_negotiation.cc:407] Connection header received
0927 02:33:56.616103 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:56.616106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:56.616155 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:56.616217 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:33:56.616481 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:56.617274 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:56.618991 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:56.619579 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:56.619694 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:56.619743 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:56.620034 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 02:33:56.620111 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 02:33:56.620234 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:56.615413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44915 (local address 127.0.98.2:48291)
0927 02:33:56.615863 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:56.615876 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:56.615993 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:56.616266 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:56.616269 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:56.616280 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:33:56.616390 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:56.616395 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:56.617366 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:56.617369 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:56.618875 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:56.618886 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:56.619701 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:56.619705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:56.619867 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:56.619901 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 02:33:56.619971 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 02:33:56.620060 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":267,"thread_start_us":103,"threads_started":1}
I20260927 02:33:56.620828 4616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f4644ceb764927a4f070b3c74f49a7" candidate_uuid: "8ef2f13ba55a4b27b870d6d8fcaf81d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b3985cfef1a481b970702e57b715f8c" is_pre_election: true
I20260927 02:33:56.621078 4882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f4644ceb764927a4f070b3c74f49a7" candidate_uuid: "8ef2f13ba55a4b27b870d6d8fcaf81d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6af42cd10de4e1fa09dee599b93cc8f" is_pre_election: true
I20260927 02:33:56.621141 4616 raft_consensus.cc:2454] T d9f4644ceb764927a4f070b3c74f49a7 P 7b3985cfef1a481b970702e57b715f8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ef2f13ba55a4b27b870d6d8fcaf81d8 in term 0.
I20260927 02:33:56.621315 4882 raft_consensus.cc:2454] T d9f4644ceb764927a4f070b3c74f49a7 P f6af42cd10de4e1fa09dee599b93cc8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ef2f13ba55a4b27b870d6d8fcaf81d8 in term 0.
I20260927 02:33:56.621587 4682 leader_election.cc:302] T d9f4644ceb764927a4f070b3c74f49a7 P 8ef2f13ba55a4b27b870d6d8fcaf81d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b3985cfef1a481b970702e57b715f8c, 8ef2f13ba55a4b27b870d6d8fcaf81d8; no voters:
I20260927 02:33:56.621863 4952 raft_consensus.cc:2781] T d9f4644ceb764927a4f070b3c74f49a7 P 8ef2f13ba55a4b27b870d6d8fcaf81d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:33:56.621938 4952 raft_consensus.cc:482] T d9f4644ceb764927a4f070b3c74f49a7 P 8ef2f13ba55a4b27b870d6d8fcaf81d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:33:56.621992 4952 raft_consensus.cc:3037] T d9f4644ceb764927a4f070b3c74f49a7 P 8ef2f13ba55a4b27b870d6d8fcaf81d8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:33:56.622967 4952 raft_consensus.cc:504] T d9f4644ceb764927a4f070b3c74f49a7 P 8ef2f13ba55a4b27b870d6d8fcaf81d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b3985cfef1a481b970702e57b715f8c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44915 } } peers { permanent_uuid: "f6af42cd10de4e1fa09dee599b93cc8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44695 } } peers { permanent_uuid: "8ef2f13ba55a4b27b870d6d8fcaf81d8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33357 } }
I20260927 02:33:56.623271 4952 leader_election.cc:288] T d9f4644ceb764927a4f070b3c74f49a7 P 8ef2f13ba55a4b27b870d6d8fcaf81d8 [CANDIDATE]: Term 1 election: Requested vote from peers 7b3985cfef1a481b970702e57b715f8c (127.0.98.1:44915), f6af42cd10de4e1fa09dee599b93cc8f (127.0.98.3:44695)
I20260927 02:33:56.623597 4882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f4644ceb764927a4f070b3c74f49a7" candidate_uuid: "8ef2f13ba55a4b27b870d6d8fcaf81d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6af42cd10de4e1fa09dee599b93cc8f"
I20260927 02:33:56.623596 4616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f4644ceb764927a4f070b3c74f49a7" candidate_uuid: "8ef2f13ba55a4b27b870d6d8fcaf81d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b3985cfef1a481b970702e57b715f8c"
I20260927 02:33:56.623710 4882 raft_consensus.cc:3037] T d9f4644ceb764927a4f070b3c74f49a7 P f6af42cd10de4e1fa09dee599b93cc8f [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:33:56.623711 4616 raft_consensus.cc:3037] T d9f4644ceb764927a4f070b3c74f49a7 P 7b3985cfef1a481b970702e57b715f8c [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:33:56.624851 4616 raft_consensus.cc:2454] T d9f4644ceb764927a4f070b3c74f49a7 P 7b3985cfef1a481b970702e57b715f8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ef2f13ba55a4b27b870d6d8fcaf81d8 in term 1.
I20260927 02:33:56.624851 4882 raft_consensus.cc:2454] T d9f4644ceb764927a4f070b3c74f49a7 P f6af42cd10de4e1fa09dee599b93cc8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ef2f13ba55a4b27b870d6d8fcaf81d8 in term 1.
I20260927 02:33:56.625144 4682 leader_election.cc:302] T d9f4644ceb764927a4f070b3c74f49a7 P 8ef2f13ba55a4b27b870d6d8fcaf81d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b3985cfef1a481b970702e57b715f8c, 8ef2f13ba55a4b27b870d6d8fcaf81d8; no voters:
I20260927 02:33:56.625303 4952 raft_consensus.cc:2781] T d9f4644ceb764927a4f070b3c74f49a7 P 8ef2f13ba55a4b27b870d6d8fcaf81d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:33:56.625509 4952 raft_consensus.cc:686] T d9f4644ceb764927a4f070b3c74f49a7 P 8ef2f13ba55a4b27b870d6d8fcaf81d8 [term 1 LEADER]: Becoming Leader. State: Replica: 8ef2f13ba55a4b27b870d6d8fcaf81d8, State: Running, Role: LEADER
I20260927 02:33:56.625803 4952 consensus_queue.cc:237] T d9f4644ceb764927a4f070b3c74f49a7 P 8ef2f13ba55a4b27b870d6d8fcaf81d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b3985cfef1a481b970702e57b715f8c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44915 } } peers { permanent_uuid: "f6af42cd10de4e1fa09dee599b93cc8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44695 } } peers { permanent_uuid: "8ef2f13ba55a4b27b870d6d8fcaf81d8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33357 } }
I20260927 02:33:56.628652 4472 catalog_manager.cc:5495] T d9f4644ceb764927a4f070b3c74f49a7 P 8ef2f13ba55a4b27b870d6d8fcaf81d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ef2f13ba55a4b27b870d6d8fcaf81d8 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "8ef2f13ba55a4b27b870d6d8fcaf81d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b3985cfef1a481b970702e57b715f8c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6af42cd10de4e1fa09dee599b93cc8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ef2f13ba55a4b27b870d6d8fcaf81d8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33357 } health_report { overall_health: HEALTHY } } }
I20260927 02:33:56.643021 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:56.639681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39698 (local address 127.0.98.2:33357)
0927 02:33:56.639798 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:56.639802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:56.639816 (+ 14us) server_negotiation.cc:407] Connection header received
0927 02:33:56.639947 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:56.639950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:56.640003 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:56.640110 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:33:56.640336 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:56.640828 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:56.641687 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:56.641899 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:56.641991 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:56.642270 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:33:56.642676 (+ 406us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:33:56.642802 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:56.642833 (+ 31us) server_negotiation.cc:299] Negotiation successful
0927 02:33:56.642885 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 02:33:56.648586 4616 raft_consensus.cc:1261] T d9f4644ceb764927a4f070b3c74f49a7 P 7b3985cfef1a481b970702e57b715f8c [term 1 FOLLOWER]: Refusing update from remote peer 8ef2f13ba55a4b27b870d6d8fcaf81d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:33:56.648582 4882 raft_consensus.cc:1261] T d9f4644ceb764927a4f070b3c74f49a7 P f6af42cd10de4e1fa09dee599b93cc8f [term 1 FOLLOWER]: Refusing update from remote peer 8ef2f13ba55a4b27b870d6d8fcaf81d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:33:56.649294 4955 consensus_queue.cc:1035] T d9f4644ceb764927a4f070b3c74f49a7 P 8ef2f13ba55a4b27b870d6d8fcaf81d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b3985cfef1a481b970702e57b715f8c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:33:56.649448 4952 consensus_queue.cc:1035] T d9f4644ceb764927a4f070b3c74f49a7 P 8ef2f13ba55a4b27b870d6d8fcaf81d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6af42cd10de4e1fa09dee599b93cc8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:33:56.655696 4959 mvcc.cc:205] Tried to move back new op lower bound from 7333791484505944064 to 7333791484421763072. Current Snapshot: MvccSnapshot[applied={T|T < 7333791484505944064}]
I20260927 02:33:56.659844 4961 mvcc.cc:205] Tried to move back new op lower bound from 7333791484505944064 to 7333791484421763072. Current Snapshot: MvccSnapshot[applied={T|T < 7333791484505944064}]
I20260927 02:33:56.664531 4469 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:33:56.664669 4469 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:33:56.666886 4469 catalog_manager.cc:5782] T 00000000000000000000000000000000 P abe60d43d20c467cbdd1fc7cecc87993: Sending DeleteTablet for 3 replicas of tablet d9f4644ceb764927a4f070b3c74f49a7
I20260927 02:33:56.667587 4596 tablet_service.cc:1514] Processing DeleteTablet for tablet d9f4644ceb764927a4f070b3c74f49a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:33:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42570
I20260927 02:33:56.667624 4729 tablet_service.cc:1514] Processing DeleteTablet for tablet d9f4644ceb764927a4f070b3c74f49a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:33:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39682
I20260927 02:33:56.667624 4862 tablet_service.cc:1514] Processing DeleteTablet for tablet d9f4644ceb764927a4f070b3c74f49a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:33:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50652
I20260927 02:33:56.668118 4970 tablet_replica.cc:330] T d9f4644ceb764927a4f070b3c74f49a7 P 7b3985cfef1a481b970702e57b715f8c: stopping tablet replica
I20260927 02:33:56.668440 4970 raft_consensus.cc:2229] T d9f4644ceb764927a4f070b3c74f49a7 P 7b3985cfef1a481b970702e57b715f8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:33:56.668468 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 4534
I20260927 02:33:56.668781 4970 raft_consensus.cc:2258] T d9f4644ceb764927a4f070b3c74f49a7 P 7b3985cfef1a481b970702e57b715f8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:33:56.669210 4969 tablet_replica.cc:330] T d9f4644ceb764927a4f070b3c74f49a7 P 8ef2f13ba55a4b27b870d6d8fcaf81d8: stopping tablet replica
I20260927 02:33:56.669339 4968 tablet_replica.cc:330] T d9f4644ceb764927a4f070b3c74f49a7 P f6af42cd10de4e1fa09dee599b93cc8f: stopping tablet replica
I20260927 02:33:56.669621 4968 raft_consensus.cc:2229] T d9f4644ceb764927a4f070b3c74f49a7 P f6af42cd10de4e1fa09dee599b93cc8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:33:56.669880 4969 raft_consensus.cc:2229] T d9f4644ceb764927a4f070b3c74f49a7 P 8ef2f13ba55a4b27b870d6d8fcaf81d8 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:33:56.670188 4968 raft_consensus.cc:2258] T d9f4644ceb764927a4f070b3c74f49a7 P f6af42cd10de4e1fa09dee599b93cc8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:33:56.670559 4969 raft_consensus.cc:2258] T d9f4644ceb764927a4f070b3c74f49a7 P 8ef2f13ba55a4b27b870d6d8fcaf81d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:33:56.672077 4969 ts_tablet_manager.cc:1905] T d9f4644ceb764927a4f070b3c74f49a7 P 8ef2f13ba55a4b27b870d6d8fcaf81d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:33:56.672084 4968 ts_tablet_manager.cc:1905] T d9f4644ceb764927a4f070b3c74f49a7 P f6af42cd10de4e1fa09dee599b93cc8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:33:56.674667 4968 ts_tablet_manager.cc:1918] T d9f4644ceb764927a4f070b3c74f49a7 P f6af42cd10de4e1fa09dee599b93cc8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:33:56.674803 4969 ts_tablet_manager.cc:1918] T d9f4644ceb764927a4f070b3c74f49a7 P 8ef2f13ba55a4b27b870d6d8fcaf81d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:33:56.674829 4968 log.cc:1196] T d9f4644ceb764927a4f070b3c74f49a7 P f6af42cd10de4e1fa09dee599b93cc8f: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-2/wal/wals/d9f4644ceb764927a4f070b3c74f49a7
I20260927 02:33:56.674975 4969 log.cc:1196] T d9f4644ceb764927a4f070b3c74f49a7 P 8ef2f13ba55a4b27b870d6d8fcaf81d8: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476410933508-392-0/minicluster-data/ts-1/wal/wals/d9f4644ceb764927a4f070b3c74f49a7
I20260927 02:33:56.675146 4968 ts_tablet_manager.cc:1939] T d9f4644ceb764927a4f070b3c74f49a7 P f6af42cd10de4e1fa09dee599b93cc8f: Deleting consensus metadata
I20260927 02:33:56.675364 4969 ts_tablet_manager.cc:1939] T d9f4644ceb764927a4f070b3c74f49a7 P 8ef2f13ba55a4b27b870d6d8fcaf81d8: Deleting consensus metadata
W20260927 02:33:56.675580 4456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44915 (error 108)
I20260927 02:33:56.676339 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 4667
I20260927 02:33:56.676394 4458 catalog_manager.cc:4841] TS f6af42cd10de4e1fa09dee599b93cc8f (127.0.98.3:44695): tablet d9f4644ceb764927a4f070b3c74f49a7 (table test-table [id=6eb40e0eafae40df87efcdf696c298d8]) successfully deleted
I20260927 02:33:56.677541 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:56.677236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44915 (local address 127.0.0.1:42576)
0927 02:33:56.677390 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:56.677461 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260927 02:33:56.677757 4456 catalog_manager.cc:4584] TS 7b3985cfef1a481b970702e57b715f8c (127.0.98.1:44915): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d9f4644ceb764927a4f070b3c74f49a7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44915: connect: Connection refused (error 111)
I20260927 02:33:56.681341 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 4800
I20260927 02:33:56.682317 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:56.681411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33357 (local address 127.0.0.1:39704)
0927 02:33:56.682200 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:56.682248 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":723}
I20260927 02:33:56.685801 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 4441
2026-09-27T02:33:56Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2079 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T02:33:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:33:56Z Disabled control of system clock
I20260927 02:33:56.712363 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32861
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 02:33:56.814316 4975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:33:56.814538 4975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:33:56.814574 4975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:33:56.818766 4975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:33:56.818823 4975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:33:56.818838 4975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:33:56.818853 4975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:33:56.818881 4975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:33:56.823612 4975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32861
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:33:56.824586 4975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:33:56.826015 4975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:33:56.833727 4981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:56.833716 4983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:56.833853 4975 server_base.cc:1034] running on GCE node
W20260927 02:33:56.833762 4980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:56.834444 4975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:33:56.835448 4975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:33:56.836604 4975 hybrid_clock.cc:648] HybridClock initialized: now 1790476436836588 us; error 26 us; skew 500 ppm
I20260927 02:33:56.838251 4975 webserver.cc:460] Webserver started at http://127.0.98.62:41007/ using document root <none> and password file <none>
I20260927 02:33:56.838768 4975 fs_manager.cc:359] Metadata directory not provided
I20260927 02:33:56.838836 4975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:33:56.839007 4975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:33:56.840901 4975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "a4fea714dfcb466aa02cc0969cdc183c"
format_stamp: "Formatted at 2026-09-27 02:33:56 on dist-test-slave-grlk"
server_key: "dd1832053d37a6308fc65203308e4f88"
server_key_iv: "d3609bb42eca0cbadff90b879a7acdfe"
server_key_version: "encryptionkey@0"
I20260927 02:33:56.841426 4975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "a4fea714dfcb466aa02cc0969cdc183c"
format_stamp: "Formatted at 2026-09-27 02:33:56 on dist-test-slave-grlk"
server_key: "dd1832053d37a6308fc65203308e4f88"
server_key_iv: "d3609bb42eca0cbadff90b879a7acdfe"
server_key_version: "encryptionkey@0"
I20260927 02:33:56.844648 4975 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:33:56.846900 4989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:56.848022 4975 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:33:56.848134 4975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "a4fea714dfcb466aa02cc0969cdc183c"
format_stamp: "Formatted at 2026-09-27 02:33:56 on dist-test-slave-grlk"
server_key: "dd1832053d37a6308fc65203308e4f88"
server_key_iv: "d3609bb42eca0cbadff90b879a7acdfe"
server_key_version: "encryptionkey@0"
I20260927 02:33:56.848219 4975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:33:56.852398 4975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:33:56.852921 4975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:33:56.853109 4975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:33:56.859112 4975 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32861
I20260927 02:33:56.859115 5041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32861 every 8 connection(s)
I20260927 02:33:56.859946 4975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:33:56.862799 5042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:33:56.867246 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 4975
I20260927 02:33:56.867358 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:33:56.867569 392 external_mini_cluster.cc:1428] Setting key f732182f171d8c1aa5ec78291aa465a2
I20260927 02:33:56.867813 5042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4fea714dfcb466aa02cc0969cdc183c: Bootstrap starting.
I20260927 02:33:56.870260 5042 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4fea714dfcb466aa02cc0969cdc183c: Neither blocks nor log segments found. Creating new log.
I20260927 02:33:56.871081 5042 log.cc:824] T 00000000000000000000000000000000 P a4fea714dfcb466aa02cc0969cdc183c: Log is configured to *not* fsync() on all Append() calls
I20260927 02:33:56.873034 5045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:56.868799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56542 (local address 127.0.98.62:32861)
0927 02:33:56.869433 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:56.869451 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:56.869489 (+ 38us) server_negotiation.cc:407] Connection header received
0927 02:33:56.870257 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:56.870280 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:56.870713 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:56.871055 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:56.871696 (+ 641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:56.871700 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:56.871710 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:33:56.871718 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:56.871764 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:33:56.871783 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:56.871875 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:56.871987 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 02:33:56.872218 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":122,"threads_started":1}
I20260927 02:33:56.873127 5042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4fea714dfcb466aa02cc0969cdc183c: No bootstrap required, opened a new log
I20260927 02:33:56.876531 5042 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4fea714dfcb466aa02cc0969cdc183c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4fea714dfcb466aa02cc0969cdc183c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32861 } }
I20260927 02:33:56.876725 5042 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4fea714dfcb466aa02cc0969cdc183c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:33:56.876768 5042 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4fea714dfcb466aa02cc0969cdc183c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4fea714dfcb466aa02cc0969cdc183c, State: Initialized, Role: FOLLOWER
I20260927 02:33:56.877110 5042 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4fea714dfcb466aa02cc0969cdc183c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4fea714dfcb466aa02cc0969cdc183c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32861 } }
I20260927 02:33:56.877220 5042 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4fea714dfcb466aa02cc0969cdc183c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:33:56.877262 5042 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4fea714dfcb466aa02cc0969cdc183c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:33:56.877324 5042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4fea714dfcb466aa02cc0969cdc183c [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:33:56.878016 5042 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4fea714dfcb466aa02cc0969cdc183c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4fea714dfcb466aa02cc0969cdc183c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32861 } }
I20260927 02:33:56.878275 5042 leader_election.cc:302] T 00000000000000000000000000000000 P a4fea714dfcb466aa02cc0969cdc183c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4fea714dfcb466aa02cc0969cdc183c; no voters:
I20260927 02:33:56.878509 5042 leader_election.cc:288] T 00000000000000000000000000000000 P a4fea714dfcb466aa02cc0969cdc183c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:33:56.878589 5047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4fea714dfcb466aa02cc0969cdc183c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:33:56.878755 5047 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4fea714dfcb466aa02cc0969cdc183c [term 1 LEADER]: Becoming Leader. State: Replica: a4fea714dfcb466aa02cc0969cdc183c, State: Running, Role: LEADER
I20260927 02:33:56.879019 5047 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4fea714dfcb466aa02cc0969cdc183c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4fea714dfcb466aa02cc0969cdc183c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32861 } }
I20260927 02:33:56.879643 5042 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4fea714dfcb466aa02cc0969cdc183c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:33:56.880066 5047 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4fea714dfcb466aa02cc0969cdc183c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4fea714dfcb466aa02cc0969cdc183c. Latest consensus state: current_term: 1 leader_uuid: "a4fea714dfcb466aa02cc0969cdc183c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4fea714dfcb466aa02cc0969cdc183c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32861 } } }
I20260927 02:33:56.880182 5047 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4fea714dfcb466aa02cc0969cdc183c [sys.catalog]: This master's current role is: LEADER
I20260927 02:33:56.880384 5047 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4fea714dfcb466aa02cc0969cdc183c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4fea714dfcb466aa02cc0969cdc183c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4fea714dfcb466aa02cc0969cdc183c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32861 } } }
I20260927 02:33:56.880522 5047 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4fea714dfcb466aa02cc0969cdc183c [sys.catalog]: This master's current role is: LEADER
I20260927 02:33:56.880563 5052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:33:56.883476 5052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:33:56.888748 5052 catalog_manager.cc:1347] Generated new cluster ID: c2c49e18771a45f2b58ffa1929b132e9
I20260927 02:33:56.888809 5052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:33:56.909400 5052 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:33:56.909693 5052 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:33:56.917689 5052 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4fea714dfcb466aa02cc0969cdc183c: Generated new TSK 0
I20260927 02:33:56.918414 5052 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:33:56.929857 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32861
--builtin_ntp_servers=127.0.98.20:44491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 02:33:57.035807 5066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:33:57.036034 5066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:33:57.036079 5066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:33:57.040020 5066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:33:57.040071 5066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:33:57.040122 5066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:33:57.044654 5066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:33:57.045594 5066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:33:57.046793 5066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:33:57.055495 5071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:57.055660 5074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:57.055496 5072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:57.055780 5066 server_base.cc:1034] running on GCE node
I20260927 02:33:57.056213 5066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:33:57.056787 5066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:33:57.057935 5066 hybrid_clock.cc:648] HybridClock initialized: now 1790476437057917 us; error 55 us; skew 500 ppm
I20260927 02:33:57.059576 5066 webserver.cc:460] Webserver started at http://127.0.98.1:43749/ using document root <none> and password file <none>
I20260927 02:33:57.060084 5066 fs_manager.cc:359] Metadata directory not provided
I20260927 02:33:57.060146 5066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:33:57.060331 5066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:33:57.062234 5066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-0/data/instance:
uuid: "775ed4ac06b24881bb9fea4501018763"
format_stamp: "Formatted at 2026-09-27 02:33:57 on dist-test-slave-grlk"
server_key: "4c9ddc2df061e928f7935db1294f8020"
server_key_iv: "4a07cac8debad9ca73694e70cd02e127"
server_key_version: "encryptionkey@0"
I20260927 02:33:57.062721 5066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-0/wal/instance:
uuid: "775ed4ac06b24881bb9fea4501018763"
format_stamp: "Formatted at 2026-09-27 02:33:57 on dist-test-slave-grlk"
server_key: "4c9ddc2df061e928f7935db1294f8020"
server_key_iv: "4a07cac8debad9ca73694e70cd02e127"
server_key_version: "encryptionkey@0"
I20260927 02:33:57.066008 5066 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:33:57.068161 5080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:57.069348 5066 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:33:57.069456 5066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-0/wal
uuid: "775ed4ac06b24881bb9fea4501018763"
format_stamp: "Formatted at 2026-09-27 02:33:57 on dist-test-slave-grlk"
server_key: "4c9ddc2df061e928f7935db1294f8020"
server_key_iv: "4a07cac8debad9ca73694e70cd02e127"
server_key_version: "encryptionkey@0"
I20260927 02:33:57.069543 5066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:33:57.073768 5066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:33:57.074295 5066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:33:57.074453 5066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:33:57.074949 5066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:33:57.075766 5066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:33:57.075811 5066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:57.075857 5066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:33:57.075886 5066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:57.083823 5066 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45985
I20260927 02:33:57.083849 5193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45985 every 8 connection(s)
I20260927 02:33:57.084599 5066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:33:57.085197 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 5066
I20260927 02:33:57.085314 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:33:57.085585 392 external_mini_cluster.cc:1428] Setting key 66b7f607da4bc302ddb9779b0365aa0a
I20260927 02:33:57.087548 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32861
--builtin_ntp_servers=127.0.98.20:44491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 02:33:57.090886 5045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:57.086461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42359 (local address 127.0.98.62:32861)
0927 02:33:57.086570 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:57.086573 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:57.087844 (+ 1271us) server_negotiation.cc:407] Connection header received
0927 02:33:57.089346 (+ 1502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:57.089349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:57.089389 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:57.089461 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:57.090491 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:57.090495 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:57.090497 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:33:57.090509 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:57.090531 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:33:57.090540 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:57.090625 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:57.090721 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 02:33:57.090761 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 02:33:57.091621 5196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:57.086957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32861 (local address 127.0.98.1:42359)
0927 02:33:57.087718 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:57.087766 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:57.089159 (+ 1393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:57.089630 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:57.089637 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:57.089720 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:33:57.090245 (+ 525us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:33:57.090247 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:33:57.090283 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:33:57.090346 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:33:57.090616 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:33:57.090621 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:33:57.090625 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:33:57.090739 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 02:33:57.090920 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":103,"threads_started":1}
I20260927 02:33:57.092682 5194 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32861
I20260927 02:33:57.092864 5194 heartbeater.cc:464] Registering TS with master...
I20260927 02:33:57.093218 5194 heartbeater.cc:511] Master 127.0.98.62:32861 requested a full tablet report, sending...
I20260927 02:33:57.094519 5006 ts_manager.cc:194] Registered new tserver with Master: 775ed4ac06b24881bb9fea4501018763 (127.0.98.1:45985)
W20260927 02:33:57.193519 5197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:33:57.193783 5197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:33:57.193828 5197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:33:57.197751 5197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:33:57.197795 5197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:33:57.197853 5197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:33:57.202389 5197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:33:57.203306 5197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:33:57.204535 5197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:33:57.213713 5205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:57.213721 5202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:57.213728 5203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:57.213977 5197 server_base.cc:1034] running on GCE node
I20260927 02:33:57.214463 5197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:33:57.215011 5197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:33:57.216162 5197 hybrid_clock.cc:648] HybridClock initialized: now 1790476437216144 us; error 29 us; skew 500 ppm
I20260927 02:33:57.217831 5197 webserver.cc:460] Webserver started at http://127.0.98.2:35663/ using document root <none> and password file <none>
I20260927 02:33:57.218340 5197 fs_manager.cc:359] Metadata directory not provided
I20260927 02:33:57.218402 5197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:33:57.218566 5197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:33:57.220404 5197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-1/data/instance:
uuid: "03936071616d47f09dc9f3b7c4eee92b"
format_stamp: "Formatted at 2026-09-27 02:33:57 on dist-test-slave-grlk"
server_key: "7ebca3b30cf7da88612e60e82b26abb7"
server_key_iv: "9c91cd3b1a3945a0fda393f0f89a4ae9"
server_key_version: "encryptionkey@0"
I20260927 02:33:57.220884 5197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-1/wal/instance:
uuid: "03936071616d47f09dc9f3b7c4eee92b"
format_stamp: "Formatted at 2026-09-27 02:33:57 on dist-test-slave-grlk"
server_key: "7ebca3b30cf7da88612e60e82b26abb7"
server_key_iv: "9c91cd3b1a3945a0fda393f0f89a4ae9"
server_key_version: "encryptionkey@0"
I20260927 02:33:57.224224 5197 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260927 02:33:57.226516 5211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:57.227568 5197 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:33:57.227672 5197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-1/wal
uuid: "03936071616d47f09dc9f3b7c4eee92b"
format_stamp: "Formatted at 2026-09-27 02:33:57 on dist-test-slave-grlk"
server_key: "7ebca3b30cf7da88612e60e82b26abb7"
server_key_iv: "9c91cd3b1a3945a0fda393f0f89a4ae9"
server_key_version: "encryptionkey@0"
I20260927 02:33:57.227759 5197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:33:57.231877 5197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:33:57.232401 5197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:33:57.232540 5197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:33:57.233033 5197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:33:57.233852 5197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:33:57.233896 5197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:57.233943 5197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:33:57.233971 5197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:57.241596 5197 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33605
I20260927 02:33:57.241647 5324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33605 every 8 connection(s)
I20260927 02:33:57.242360 5197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:33:57.243254 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 5197
I20260927 02:33:57.243363 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:33:57.243621 392 external_mini_cluster.cc:1428] Setting key 5496899926ddf0a24b044ac2010c819d
I20260927 02:33:57.245363 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32861
--builtin_ntp_servers=127.0.98.20:44491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 02:33:57.247902 5045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:57.243915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41071 (local address 127.0.98.62:32861)
0927 02:33:57.244031 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:57.244034 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:57.245162 (+ 1128us) server_negotiation.cc:407] Connection header received
0927 02:33:57.246387 (+ 1225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:57.246390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:57.246430 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:57.246499 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:57.247497 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:57.247500 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:57.247503 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:33:57.247515 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:57.247535 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:33:57.247544 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:57.247617 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:57.247744 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 02:33:57.247783 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 02:33:57.248778 5327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:57.244347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32861 (local address 127.0.98.2:41071)
0927 02:33:57.245018 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:57.245063 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:57.246201 (+ 1138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:57.246664 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:57.246671 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:57.246750 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:33:57.247274 (+ 524us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:33:57.247276 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:33:57.247311 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:33:57.247371 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:33:57.247616 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:33:57.247624 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:33:57.247629 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:33:57.247765 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 02:33:57.247937 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":96,"threads_started":1}
I20260927 02:33:57.249857 5325 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32861
I20260927 02:33:57.250052 5325 heartbeater.cc:464] Registering TS with master...
I20260927 02:33:57.250385 5325 heartbeater.cc:511] Master 127.0.98.62:32861 requested a full tablet report, sending...
I20260927 02:33:57.251294 5006 ts_manager.cc:194] Registered new tserver with Master: 03936071616d47f09dc9f3b7c4eee92b (127.0.98.2:33605)
W20260927 02:33:57.347937 5328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:33:57.348177 5328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:33:57.348229 5328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:33:57.352250 5328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:33:57.352298 5328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:33:57.352370 5328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:33:57.356984 5328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:33:57.357919 5328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:33:57.359162 5328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:33:57.367727 5333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:57.367799 5328 server_base.cc:1034] running on GCE node
W20260927 02:33:57.367735 5334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:57.367942 5336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:57.368288 5328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:33:57.368865 5328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:33:57.370060 5328 hybrid_clock.cc:648] HybridClock initialized: now 1790476437370045 us; error 29 us; skew 500 ppm
I20260927 02:33:57.371721 5328 webserver.cc:460] Webserver started at http://127.0.98.3:39633/ using document root <none> and password file <none>
I20260927 02:33:57.372293 5328 fs_manager.cc:359] Metadata directory not provided
I20260927 02:33:57.372376 5328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:33:57.372566 5328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:33:57.374526 5328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-2/data/instance:
uuid: "f48755ea8ba846a2967f811912e584e3"
format_stamp: "Formatted at 2026-09-27 02:33:57 on dist-test-slave-grlk"
server_key: "989f4df5e458651ed850c174ebf1c753"
server_key_iv: "e655deac6c82c0eab86a564fc6f5d1c7"
server_key_version: "encryptionkey@0"
I20260927 02:33:57.375011 5328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f48755ea8ba846a2967f811912e584e3"
format_stamp: "Formatted at 2026-09-27 02:33:57 on dist-test-slave-grlk"
server_key: "989f4df5e458651ed850c174ebf1c753"
server_key_iv: "e655deac6c82c0eab86a564fc6f5d1c7"
server_key_version: "encryptionkey@0"
I20260927 02:33:57.378423 5328 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 02:33:57.380609 5342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:57.381848 5328 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:33:57.381958 5328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-2/wal
uuid: "f48755ea8ba846a2967f811912e584e3"
format_stamp: "Formatted at 2026-09-27 02:33:57 on dist-test-slave-grlk"
server_key: "989f4df5e458651ed850c174ebf1c753"
server_key_iv: "e655deac6c82c0eab86a564fc6f5d1c7"
server_key_version: "encryptionkey@0"
I20260927 02:33:57.382064 5328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:33:57.386183 5328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:33:57.386714 5328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:33:57.386854 5328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:33:57.387354 5328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:33:57.388167 5328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:33:57.388211 5328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:57.388259 5328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:33:57.388284 5328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:57.395907 5328 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41679
I20260927 02:33:57.395931 5455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41679 every 8 connection(s)
I20260927 02:33:57.396680 5328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:33:57.400803 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 5328
I20260927 02:33:57.400903 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:33:57.401151 392 external_mini_cluster.cc:1428] Setting key b2b567dfce724f34f27aeb5ec1dbed79
I20260927 02:33:57.402222 5045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:57.398290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47745 (local address 127.0.98.62:32861)
0927 02:33:57.398412 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:57.398415 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:57.399349 (+ 934us) server_negotiation.cc:407] Connection header received
0927 02:33:57.400560 (+ 1211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:57.400564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:57.400604 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:57.400688 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:57.401755 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:57.401758 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:57.401760 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:33:57.401772 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:57.401792 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:33:57.401803 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:57.401859 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:57.402056 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 02:33:57.402097 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 02:33:57.403174 5458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:57.398641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32861 (local address 127.0.98.3:47745)
0927 02:33:57.399226 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:57.399260 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:57.400376 (+ 1116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:57.400837 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:57.400844 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:57.400970 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:33:57.401496 (+ 526us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:33:57.401498 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:33:57.401534 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:33:57.401596 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:33:57.401922 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:33:57.401930 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:33:57.401934 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:33:57.402077 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 02:33:57.402301 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":97,"threads_started":1}
I20260927 02:33:57.404456 5456 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32861
I20260927 02:33:57.404676 5456 heartbeater.cc:464] Registering TS with master...
I20260927 02:33:57.405097 5456 heartbeater.cc:511] Master 127.0.98.62:32861 requested a full tablet report, sending...
I20260927 02:33:57.405990 5006 ts_manager.cc:194] Registered new tserver with Master: f48755ea8ba846a2967f811912e584e3 (127.0.98.3:41679)
I20260927 02:33:57.415083 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:33:57.418133 5045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:57.416769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56552 (local address 127.0.98.62:32861)
0927 02:33:57.416895 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:57.416899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:57.417054 (+ 155us) server_negotiation.cc:407] Connection header received
0927 02:33:57.417157 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:57.417159 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:57.417190 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:57.417248 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:57.417834 (+ 586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:57.417837 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:57.417839 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:33:57.417848 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:57.417870 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:33:57.417881 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:57.417940 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:57.418005 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 02:33:57.418034 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 02:33:57.420384 5006 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56552:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:33:57.423141 5006 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:33:57.433648 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:57.430534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41679 (local address 127.0.0.1:49778)
0927 02:33:57.431240 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:57.431260 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:57.431359 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:57.432062 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:57.432068 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:57.432141 (+ 73us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:33:57.432641 (+ 500us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:33:57.432643 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:33:57.432662 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:33:57.432725 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:33:57.433413 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:33:57.433416 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:33:57.433418 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 02:33:57.433466 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 02:33:57.433498 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":92,"threads_started":1}
I20260927 02:33:57.433823 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:57.431651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57020 (local address 127.0.98.2:33605)
0927 02:33:57.431907 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:57.431916 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:57.431935 (+ 19us) server_negotiation.cc:407] Connection header received
0927 02:33:57.431980 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:57.431985 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:57.432154 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:57.432460 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:57.433063 (+ 603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:57.433069 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:57.433080 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:33:57.433090 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:57.433129 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:33:57.433155 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:57.433293 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:57.433436 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 02:33:57.433504 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":79,"threads_started":1}
I20260927 02:33:57.433837 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:57.430386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45985 (local address 127.0.0.1:53860)
0927 02:33:57.431184 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:57.431796 (+ 612us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:57.431861 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:57.432829 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:57.432832 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:57.432844 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:33:57.433268 (+ 424us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:33:57.433270 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:33:57.433279 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:33:57.433301 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:33:57.433627 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:33:57.433630 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:33:57.433633 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:33:57.433712 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 02:33:57.433741 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":64,"threads_started":1}
I20260927 02:33:57.434156 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:57.430517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53860 (local address 127.0.98.1:45985)
0927 02:33:57.431751 (+ 1234us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:57.431762 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:57.431932 (+ 170us) server_negotiation.cc:407] Connection header received
0927 02:33:57.431986 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:57.431990 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:57.432156 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:57.432267 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:57.433405 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:57.433411 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:57.433421 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:33:57.433432 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:57.433469 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:33:57.433499 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:57.433590 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:57.433925 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 02:33:57.433987 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1141,"thread_start_us":112,"threads_started":1}
I20260927 02:33:57.433660 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:57.430952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33605 (local address 127.0.0.1:57020)
0927 02:33:57.431466 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:57.431478 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:57.431539 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:57.432295 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:57.432298 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:57.432310 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:33:57.432779 (+ 469us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:33:57.432781 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:33:57.432791 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:33:57.432817 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:33:57.433288 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:33:57.433297 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:33:57.433303 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 02:33:57.433412 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 02:33:57.433475 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":90,"threads_started":1}
I20260927 02:33:57.434572 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:57.431322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49778 (local address 127.0.98.3:41679)
0927 02:33:57.431603 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:57.431615 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:57.431652 (+ 37us) server_negotiation.cc:407] Connection header received
0927 02:33:57.431703 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:57.431709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:57.431909 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:57.432045 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:57.433073 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:57.433078 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:57.433086 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:33:57.433096 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:57.433129 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:33:57.433155 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:57.434326 (+ 1171us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:57.434368 (+ 42us) server_negotiation.cc:299] Negotiation successful
0927 02:33:57.434420 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":102,"threads_started":1}
I20260927 02:33:57.436689 5128 tablet_service.cc:1467] Processing CreateTablet for tablet 31cc36d9707c4e6697593d62d53487ed (DEFAULT_TABLE table=test-table [id=76e0967b71bf4e50b507ac30f2345550]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:33:57.436887 5390 tablet_service.cc:1467] Processing CreateTablet for tablet 31cc36d9707c4e6697593d62d53487ed (DEFAULT_TABLE table=test-table [id=76e0967b71bf4e50b507ac30f2345550]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:33:57.436887 5259 tablet_service.cc:1467] Processing CreateTablet for tablet 31cc36d9707c4e6697593d62d53487ed (DEFAULT_TABLE table=test-table [id=76e0967b71bf4e50b507ac30f2345550]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:33:57.437758 5128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31cc36d9707c4e6697593d62d53487ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:33:57.437827 5259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31cc36d9707c4e6697593d62d53487ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:33:57.437827 5390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31cc36d9707c4e6697593d62d53487ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:33:57.443313 5473 tablet_bootstrap.cc:492] T 31cc36d9707c4e6697593d62d53487ed P 775ed4ac06b24881bb9fea4501018763: Bootstrap starting.
I20260927 02:33:57.444685 5474 tablet_bootstrap.cc:492] T 31cc36d9707c4e6697593d62d53487ed P 03936071616d47f09dc9f3b7c4eee92b: Bootstrap starting.
I20260927 02:33:57.445209 5475 tablet_bootstrap.cc:492] T 31cc36d9707c4e6697593d62d53487ed P f48755ea8ba846a2967f811912e584e3: Bootstrap starting.
I20260927 02:33:57.445602 5473 tablet_bootstrap.cc:654] T 31cc36d9707c4e6697593d62d53487ed P 775ed4ac06b24881bb9fea4501018763: Neither blocks nor log segments found. Creating new log.
I20260927 02:33:57.446344 5473 log.cc:824] T 31cc36d9707c4e6697593d62d53487ed P 775ed4ac06b24881bb9fea4501018763: Log is configured to *not* fsync() on all Append() calls
I20260927 02:33:57.446655 5474 tablet_bootstrap.cc:654] T 31cc36d9707c4e6697593d62d53487ed P 03936071616d47f09dc9f3b7c4eee92b: Neither blocks nor log segments found. Creating new log.
I20260927 02:33:57.447161 5475 tablet_bootstrap.cc:654] T 31cc36d9707c4e6697593d62d53487ed P f48755ea8ba846a2967f811912e584e3: Neither blocks nor log segments found. Creating new log.
I20260927 02:33:57.447445 5474 log.cc:824] T 31cc36d9707c4e6697593d62d53487ed P 03936071616d47f09dc9f3b7c4eee92b: Log is configured to *not* fsync() on all Append() calls
I20260927 02:33:57.447923 5475 log.cc:824] T 31cc36d9707c4e6697593d62d53487ed P f48755ea8ba846a2967f811912e584e3: Log is configured to *not* fsync() on all Append() calls
I20260927 02:33:57.448069 5473 tablet_bootstrap.cc:492] T 31cc36d9707c4e6697593d62d53487ed P 775ed4ac06b24881bb9fea4501018763: No bootstrap required, opened a new log
I20260927 02:33:57.448252 5473 ts_tablet_manager.cc:1397] T 31cc36d9707c4e6697593d62d53487ed P 775ed4ac06b24881bb9fea4501018763: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 02:33:57.449214 5474 tablet_bootstrap.cc:492] T 31cc36d9707c4e6697593d62d53487ed P 03936071616d47f09dc9f3b7c4eee92b: No bootstrap required, opened a new log
I20260927 02:33:57.449384 5474 ts_tablet_manager.cc:1397] T 31cc36d9707c4e6697593d62d53487ed P 03936071616d47f09dc9f3b7c4eee92b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 02:33:57.449566 5475 tablet_bootstrap.cc:492] T 31cc36d9707c4e6697593d62d53487ed P f48755ea8ba846a2967f811912e584e3: No bootstrap required, opened a new log
I20260927 02:33:57.449731 5475 ts_tablet_manager.cc:1397] T 31cc36d9707c4e6697593d62d53487ed P f48755ea8ba846a2967f811912e584e3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 02:33:57.451865 5473 raft_consensus.cc:348] T 31cc36d9707c4e6697593d62d53487ed P 775ed4ac06b24881bb9fea4501018763 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "775ed4ac06b24881bb9fea4501018763" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45985 } } peers { permanent_uuid: "f48755ea8ba846a2967f811912e584e3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41679 } } peers { permanent_uuid: "03936071616d47f09dc9f3b7c4eee92b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33605 } }
I20260927 02:33:57.452073 5473 raft_consensus.cc:374] T 31cc36d9707c4e6697593d62d53487ed P 775ed4ac06b24881bb9fea4501018763 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:33:57.452124 5473 raft_consensus.cc:729] T 31cc36d9707c4e6697593d62d53487ed P 775ed4ac06b24881bb9fea4501018763 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 775ed4ac06b24881bb9fea4501018763, State: Initialized, Role: FOLLOWER
I20260927 02:33:57.452553 5473 consensus_queue.cc:260] T 31cc36d9707c4e6697593d62d53487ed P 775ed4ac06b24881bb9fea4501018763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "775ed4ac06b24881bb9fea4501018763" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45985 } } peers { permanent_uuid: "f48755ea8ba846a2967f811912e584e3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41679 } } peers { permanent_uuid: "03936071616d47f09dc9f3b7c4eee92b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33605 } }
I20260927 02:33:57.453154 5194 heartbeater.cc:503] Master 127.0.98.62:32861 was elected leader, sending a full tablet report...
I20260927 02:33:57.453472 5473 ts_tablet_manager.cc:1428] T 31cc36d9707c4e6697593d62d53487ed P 775ed4ac06b24881bb9fea4501018763: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 02:33:57.453464 5474 raft_consensus.cc:348] T 31cc36d9707c4e6697593d62d53487ed P 03936071616d47f09dc9f3b7c4eee92b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "775ed4ac06b24881bb9fea4501018763" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45985 } } peers { permanent_uuid: "f48755ea8ba846a2967f811912e584e3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41679 } } peers { permanent_uuid: "03936071616d47f09dc9f3b7c4eee92b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33605 } }
I20260927 02:33:57.453701 5474 raft_consensus.cc:374] T 31cc36d9707c4e6697593d62d53487ed P 03936071616d47f09dc9f3b7c4eee92b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:33:57.453756 5474 raft_consensus.cc:729] T 31cc36d9707c4e6697593d62d53487ed P 03936071616d47f09dc9f3b7c4eee92b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03936071616d47f09dc9f3b7c4eee92b, State: Initialized, Role: FOLLOWER
I20260927 02:33:57.454126 5474 consensus_queue.cc:260] T 31cc36d9707c4e6697593d62d53487ed P 03936071616d47f09dc9f3b7c4eee92b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "775ed4ac06b24881bb9fea4501018763" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45985 } } peers { permanent_uuid: "f48755ea8ba846a2967f811912e584e3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41679 } } peers { permanent_uuid: "03936071616d47f09dc9f3b7c4eee92b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33605 } }
I20260927 02:33:57.454397 5475 raft_consensus.cc:348] T 31cc36d9707c4e6697593d62d53487ed P f48755ea8ba846a2967f811912e584e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "775ed4ac06b24881bb9fea4501018763" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45985 } } peers { permanent_uuid: "f48755ea8ba846a2967f811912e584e3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41679 } } peers { permanent_uuid: "03936071616d47f09dc9f3b7c4eee92b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33605 } }
I20260927 02:33:57.454627 5475 raft_consensus.cc:374] T 31cc36d9707c4e6697593d62d53487ed P f48755ea8ba846a2967f811912e584e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:33:57.454684 5475 raft_consensus.cc:729] T 31cc36d9707c4e6697593d62d53487ed P f48755ea8ba846a2967f811912e584e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f48755ea8ba846a2967f811912e584e3, State: Initialized, Role: FOLLOWER
I20260927 02:33:57.454787 5325 heartbeater.cc:503] Master 127.0.98.62:32861 was elected leader, sending a full tablet report...
I20260927 02:33:57.455000 5474 ts_tablet_manager.cc:1428] T 31cc36d9707c4e6697593d62d53487ed P 03936071616d47f09dc9f3b7c4eee92b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 02:33:57.455066 5475 consensus_queue.cc:260] T 31cc36d9707c4e6697593d62d53487ed P f48755ea8ba846a2967f811912e584e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "775ed4ac06b24881bb9fea4501018763" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45985 } } peers { permanent_uuid: "f48755ea8ba846a2967f811912e584e3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41679 } } peers { permanent_uuid: "03936071616d47f09dc9f3b7c4eee92b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33605 } }
I20260927 02:33:57.455628 5456 heartbeater.cc:503] Master 127.0.98.62:32861 was elected leader, sending a full tablet report...
I20260927 02:33:57.455946 5475 ts_tablet_manager.cc:1428] T 31cc36d9707c4e6697593d62d53487ed P f48755ea8ba846a2967f811912e584e3: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 02:33:57.456195 5480 raft_consensus.cc:482] T 31cc36d9707c4e6697593d62d53487ed P 03936071616d47f09dc9f3b7c4eee92b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:33:57.456321 5480 raft_consensus.cc:504] T 31cc36d9707c4e6697593d62d53487ed P 03936071616d47f09dc9f3b7c4eee92b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "775ed4ac06b24881bb9fea4501018763" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45985 } } peers { permanent_uuid: "f48755ea8ba846a2967f811912e584e3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41679 } } peers { permanent_uuid: "03936071616d47f09dc9f3b7c4eee92b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33605 } }
I20260927 02:33:57.459542 5480 leader_election.cc:288] T 31cc36d9707c4e6697593d62d53487ed P 03936071616d47f09dc9f3b7c4eee92b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 775ed4ac06b24881bb9fea4501018763 (127.0.98.1:45985), f48755ea8ba846a2967f811912e584e3 (127.0.98.3:41679)
I20260927 02:33:57.460561 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:57.457598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35269 (local address 127.0.98.1:45985)
0927 02:33:57.457697 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:57.457702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:57.459192 (+ 1490us) server_negotiation.cc:407] Connection header received
0927 02:33:57.459230 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:57.459233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:57.459268 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:57.459337 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:57.460197 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:57.460200 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:57.460203 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:33:57.460215 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:57.460253 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:33:57.460263 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:57.460334 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:57.460414 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 02:33:57.460448 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 02:33:57.460567 5327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:57.457535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45985 (local address 127.0.98.2:35269)
0927 02:33:57.459001 (+ 1466us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:57.459016 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:57.459112 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:57.459390 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:57.459393 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:57.459410 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:33:57.460036 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:33:57.460039 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:33:57.460052 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:33:57.460082 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:33:57.460332 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:33:57.460336 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:33:57.460339 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:33:57.460416 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 02:33:57.460448 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1362}
I20260927 02:33:57.461593 5148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31cc36d9707c4e6697593d62d53487ed" candidate_uuid: "03936071616d47f09dc9f3b7c4eee92b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "775ed4ac06b24881bb9fea4501018763" is_pre_election: true
I20260927 02:33:57.461833 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:57.459737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41679 (local address 127.0.98.2:55253)
0927 02:33:57.460023 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:57.460035 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:57.460149 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:57.460784 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:57.460787 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:57.460801 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:33:57.461257 (+ 456us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:33:57.461259 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:33:57.461270 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:33:57.461297 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:33:57.461600 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:33:57.461603 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:33:57.461605 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 02:33:57.461673 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 02:33:57.461706 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":75,"threads_started":1}
I20260927 02:33:57.461855 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:57.459808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55253 (local address 127.0.98.3:41679)
0927 02:33:57.459904 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:57.459908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:57.460101 (+ 193us) server_negotiation.cc:407] Connection header received
0927 02:33:57.460232 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:57.460235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:57.460270 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:57.460337 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:57.461425 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:57.461428 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:57.461431 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:33:57.461443 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:57.461462 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:33:57.461470 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:57.461523 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:57.461704 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 02:33:57.461746 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 02:33:57.461840 5148 raft_consensus.cc:2454] T 31cc36d9707c4e6697593d62d53487ed P 775ed4ac06b24881bb9fea4501018763 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03936071616d47f09dc9f3b7c4eee92b in term 0.
I20260927 02:33:57.462386 5410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31cc36d9707c4e6697593d62d53487ed" candidate_uuid: "03936071616d47f09dc9f3b7c4eee92b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f48755ea8ba846a2967f811912e584e3" is_pre_election: true
I20260927 02:33:57.462430 5214 leader_election.cc:302] T 31cc36d9707c4e6697593d62d53487ed P 03936071616d47f09dc9f3b7c4eee92b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03936071616d47f09dc9f3b7c4eee92b, 775ed4ac06b24881bb9fea4501018763; no voters:
I20260927 02:33:57.462612 5410 raft_consensus.cc:2454] T 31cc36d9707c4e6697593d62d53487ed P f48755ea8ba846a2967f811912e584e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03936071616d47f09dc9f3b7c4eee92b in term 0.
I20260927 02:33:57.462666 5480 raft_consensus.cc:2781] T 31cc36d9707c4e6697593d62d53487ed P 03936071616d47f09dc9f3b7c4eee92b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:33:57.462924 5480 raft_consensus.cc:482] T 31cc36d9707c4e6697593d62d53487ed P 03936071616d47f09dc9f3b7c4eee92b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:33:57.463022 5480 raft_consensus.cc:3037] T 31cc36d9707c4e6697593d62d53487ed P 03936071616d47f09dc9f3b7c4eee92b [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:33:57.464216 5480 raft_consensus.cc:504] T 31cc36d9707c4e6697593d62d53487ed P 03936071616d47f09dc9f3b7c4eee92b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "775ed4ac06b24881bb9fea4501018763" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45985 } } peers { permanent_uuid: "f48755ea8ba846a2967f811912e584e3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41679 } } peers { permanent_uuid: "03936071616d47f09dc9f3b7c4eee92b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33605 } }
I20260927 02:33:57.464565 5480 leader_election.cc:288] T 31cc36d9707c4e6697593d62d53487ed P 03936071616d47f09dc9f3b7c4eee92b [CANDIDATE]: Term 1 election: Requested vote from peers 775ed4ac06b24881bb9fea4501018763 (127.0.98.1:45985), f48755ea8ba846a2967f811912e584e3 (127.0.98.3:41679)
I20260927 02:33:57.464805 5148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31cc36d9707c4e6697593d62d53487ed" candidate_uuid: "03936071616d47f09dc9f3b7c4eee92b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "775ed4ac06b24881bb9fea4501018763"
I20260927 02:33:57.464890 5410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31cc36d9707c4e6697593d62d53487ed" candidate_uuid: "03936071616d47f09dc9f3b7c4eee92b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f48755ea8ba846a2967f811912e584e3"
I20260927 02:33:57.464918 5148 raft_consensus.cc:3037] T 31cc36d9707c4e6697593d62d53487ed P 775ed4ac06b24881bb9fea4501018763 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:33:57.465018 5410 raft_consensus.cc:3037] T 31cc36d9707c4e6697593d62d53487ed P f48755ea8ba846a2967f811912e584e3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:33:57.466136 5148 raft_consensus.cc:2454] T 31cc36d9707c4e6697593d62d53487ed P 775ed4ac06b24881bb9fea4501018763 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03936071616d47f09dc9f3b7c4eee92b in term 1.
I20260927 02:33:57.466198 5410 raft_consensus.cc:2454] T 31cc36d9707c4e6697593d62d53487ed P f48755ea8ba846a2967f811912e584e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03936071616d47f09dc9f3b7c4eee92b in term 1.
I20260927 02:33:57.466379 5214 leader_election.cc:302] T 31cc36d9707c4e6697593d62d53487ed P 03936071616d47f09dc9f3b7c4eee92b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03936071616d47f09dc9f3b7c4eee92b, 775ed4ac06b24881bb9fea4501018763; no voters:
I20260927 02:33:57.466513 5480 raft_consensus.cc:2781] T 31cc36d9707c4e6697593d62d53487ed P 03936071616d47f09dc9f3b7c4eee92b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:33:57.466686 5480 raft_consensus.cc:686] T 31cc36d9707c4e6697593d62d53487ed P 03936071616d47f09dc9f3b7c4eee92b [term 1 LEADER]: Becoming Leader. State: Replica: 03936071616d47f09dc9f3b7c4eee92b, State: Running, Role: LEADER
I20260927 02:33:57.466964 5480 consensus_queue.cc:237] T 31cc36d9707c4e6697593d62d53487ed P 03936071616d47f09dc9f3b7c4eee92b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "775ed4ac06b24881bb9fea4501018763" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45985 } } peers { permanent_uuid: "f48755ea8ba846a2967f811912e584e3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41679 } } peers { permanent_uuid: "03936071616d47f09dc9f3b7c4eee92b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33605 } }
I20260927 02:33:57.469923 5006 catalog_manager.cc:5495] T 31cc36d9707c4e6697593d62d53487ed P 03936071616d47f09dc9f3b7c4eee92b reported cstate change: term changed from 0 to 1, leader changed from <none> to 03936071616d47f09dc9f3b7c4eee92b (127.0.98.2). New cstate: current_term: 1 leader_uuid: "03936071616d47f09dc9f3b7c4eee92b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "775ed4ac06b24881bb9fea4501018763" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f48755ea8ba846a2967f811912e584e3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03936071616d47f09dc9f3b7c4eee92b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33605 } health_report { overall_health: HEALTHY } } }
I20260927 02:33:57.481851 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:57.480599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57026 (local address 127.0.98.2:33605)
0927 02:33:57.480709 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:57.480713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:57.480726 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:33:57.480835 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:57.480837 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:57.480890 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:57.481002 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:57.481529 (+ 527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:57.481532 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:57.481534 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:33:57.481543 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:57.481561 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:33:57.481568 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:57.481634 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:57.481724 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 02:33:57.481756 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 02:33:57.487134 5410 raft_consensus.cc:1261] T 31cc36d9707c4e6697593d62d53487ed P f48755ea8ba846a2967f811912e584e3 [term 1 FOLLOWER]: Refusing update from remote peer 03936071616d47f09dc9f3b7c4eee92b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:33:57.487134 5148 raft_consensus.cc:1261] T 31cc36d9707c4e6697593d62d53487ed P 775ed4ac06b24881bb9fea4501018763 [term 1 FOLLOWER]: Refusing update from remote peer 03936071616d47f09dc9f3b7c4eee92b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:33:57.487771 5483 consensus_queue.cc:1035] T 31cc36d9707c4e6697593d62d53487ed P 03936071616d47f09dc9f3b7c4eee92b [LEADER]: Connected to new peer: Peer: permanent_uuid: "775ed4ac06b24881bb9fea4501018763" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:33:57.487926 5480 consensus_queue.cc:1035] T 31cc36d9707c4e6697593d62d53487ed P 03936071616d47f09dc9f3b7c4eee92b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f48755ea8ba846a2967f811912e584e3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 02:33:57.493098 5326 tablet.cc:2369] T 31cc36d9707c4e6697593d62d53487ed P 03936071616d47f09dc9f3b7c4eee92b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:33:57.496153 5486 mvcc.cc:205] Tried to move back new op lower bound from 7333791487940956160 to 7333791487867129856. Current Snapshot: MvccSnapshot[applied={T|T < 7333791487940956160}]
I20260927 02:33:57.498075 5487 mvcc.cc:205] Tried to move back new op lower bound from 7333791487940956160 to 7333791487867129856. Current Snapshot: MvccSnapshot[applied={T|T < 7333791487940956160}]
I20260927 02:33:57.503360 5005 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:33:57.503496 5005 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:33:57.505527 5005 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a4fea714dfcb466aa02cc0969cdc183c: Sending DeleteTablet for 3 replicas of tablet 31cc36d9707c4e6697593d62d53487ed
I20260927 02:33:57.506316 5128 tablet_service.cc:1514] Processing DeleteTablet for tablet 31cc36d9707c4e6697593d62d53487ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:33:57 UTC) from {username='slave'} at 127.0.0.1:53860
I20260927 02:33:57.506335 5390 tablet_service.cc:1514] Processing DeleteTablet for tablet 31cc36d9707c4e6697593d62d53487ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:33:57 UTC) from {username='slave'} at 127.0.0.1:49778
I20260927 02:33:57.506354 5259 tablet_service.cc:1514] Processing DeleteTablet for tablet 31cc36d9707c4e6697593d62d53487ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:33:57 UTC) from {username='slave'} at 127.0.0.1:57020
I20260927 02:33:57.506855 5498 tablet_replica.cc:330] T 31cc36d9707c4e6697593d62d53487ed P 03936071616d47f09dc9f3b7c4eee92b: stopping tablet replica
I20260927 02:33:57.506856 5497 tablet_replica.cc:330] T 31cc36d9707c4e6697593d62d53487ed P 775ed4ac06b24881bb9fea4501018763: stopping tablet replica
I20260927 02:33:57.507021 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 5066
I20260927 02:33:57.507938 5496 tablet_replica.cc:330] T 31cc36d9707c4e6697593d62d53487ed P f48755ea8ba846a2967f811912e584e3: stopping tablet replica
I20260927 02:33:57.508181 5496 raft_consensus.cc:2229] T 31cc36d9707c4e6697593d62d53487ed P f48755ea8ba846a2967f811912e584e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:33:57.508424 5496 raft_consensus.cc:2258] T 31cc36d9707c4e6697593d62d53487ed P f48755ea8ba846a2967f811912e584e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:33:57.508752 5498 raft_consensus.cc:2229] T 31cc36d9707c4e6697593d62d53487ed P 03936071616d47f09dc9f3b7c4eee92b [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:33:57.509459 5498 raft_consensus.cc:2258] T 31cc36d9707c4e6697593d62d53487ed P 03936071616d47f09dc9f3b7c4eee92b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:33:57.509987 5496 ts_tablet_manager.cc:1905] T 31cc36d9707c4e6697593d62d53487ed P f48755ea8ba846a2967f811912e584e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:33:57.510767 5498 ts_tablet_manager.cc:1905] T 31cc36d9707c4e6697593d62d53487ed P 03936071616d47f09dc9f3b7c4eee92b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:33:57.512501 5496 ts_tablet_manager.cc:1918] T 31cc36d9707c4e6697593d62d53487ed P f48755ea8ba846a2967f811912e584e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:33:57.512593 5496 log.cc:1196] T 31cc36d9707c4e6697593d62d53487ed P f48755ea8ba846a2967f811912e584e3: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-2/wal/wals/31cc36d9707c4e6697593d62d53487ed
I20260927 02:33:57.512871 5496 ts_tablet_manager.cc:1939] T 31cc36d9707c4e6697593d62d53487ed P f48755ea8ba846a2967f811912e584e3: Deleting consensus metadata
I20260927 02:33:57.513586 5498 ts_tablet_manager.cc:1918] T 31cc36d9707c4e6697593d62d53487ed P 03936071616d47f09dc9f3b7c4eee92b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:33:57.513739 5498 log.cc:1196] T 31cc36d9707c4e6697593d62d53487ed P 03936071616d47f09dc9f3b7c4eee92b: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476410933508-392-0/minicluster-data/ts-1/wal/wals/31cc36d9707c4e6697593d62d53487ed
I20260927 02:33:57.513994 4993 catalog_manager.cc:4841] TS f48755ea8ba846a2967f811912e584e3 (127.0.98.3:41679): tablet 31cc36d9707c4e6697593d62d53487ed (table test-table [id=76e0967b71bf4e50b507ac30f2345550]) successfully deleted
I20260927 02:33:57.514074 5498 ts_tablet_manager.cc:1939] T 31cc36d9707c4e6697593d62d53487ed P 03936071616d47f09dc9f3b7c4eee92b: Deleting consensus metadata
W20260927 02:33:57.514480 4992 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45985 (error 108)
I20260927 02:33:57.514674 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 5197
I20260927 02:33:57.516760 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:57.516505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45985 (local address 127.0.0.1:53862)
0927 02:33:57.516627 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:57.516687 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 02:33:57.516886 4992 catalog_manager.cc:4584] TS 775ed4ac06b24881bb9fea4501018763 (127.0.98.1:45985): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 31cc36d9707c4e6697593d62d53487ed: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45985: connect: Connection refused (error 111)
I20260927 02:33:57.519227 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 5328
I20260927 02:33:57.520853 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:57.519305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33605 (local address 127.0.0.1:57032)
0927 02:33:57.520730 (+ 1425us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:57.520783 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:33605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1348}
I20260927 02:33:57.523787 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 4975
2026-09-27T02:33:57Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (837 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 02:33:57.542748 392 mini_oidc.cc:152] Starting JWKS server
I20260927 02:33:57.544358 392 webserver.cc:460] Webserver started at https://127.0.0.1:36355/ using document root <none> and password file <none>
I20260927 02:33:57.544526 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 02:33:57.544788 392 webserver.cc:460] Webserver started at http://127.0.0.1:37759/ using document root <none> and password file <none>
2026-09-27T02:33:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:33:57Z Disabled control of system clock
I20260927 02:33:57.552901 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34701
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34701
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36355/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476410933508-392-0/testchainca.cert with env {}
W20260927 02:33:57.655730 5505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:33:57.655964 5505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:33:57.655999 5505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:33:57.659870 5505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:33:57.659917 5505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:33:57.659931 5505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:33:57.659946 5505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:33:57.659960 5505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 02:33:57.659988 5505 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 02:33:57.660001 5505 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36355/jwks.json
I20260927 02:33:57.664994 5505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34701
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36355/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476410933508-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:33:57.665851 5505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:33:57.666708 5505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:33:57.674368 5511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:57.674364 5510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:57.674393 5513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:57.674636 5505 server_base.cc:1034] running on GCE node
I20260927 02:33:57.675175 5505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:33:57.676172 5505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:33:57.677348 5505 hybrid_clock.cc:648] HybridClock initialized: now 1790476437677334 us; error 27 us; skew 500 ppm
I20260927 02:33:57.678984 5505 webserver.cc:460] Webserver started at http://127.0.98.62:45073/ using document root <none> and password file <none>
I20260927 02:33:57.679487 5505 fs_manager.cc:359] Metadata directory not provided
I20260927 02:33:57.679548 5505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:33:57.679725 5505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:33:57.681653 5505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "5976a3fb096742fd81a53559e976bb22"
format_stamp: "Formatted at 2026-09-27 02:33:57 on dist-test-slave-grlk"
server_key: "99ced2762e58a819cb4e4ca54fc417eb"
server_key_iv: "da3b297856eb3d7a6b7e929135cf4498"
server_key_version: "encryptionkey@0"
I20260927 02:33:57.682163 5505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "5976a3fb096742fd81a53559e976bb22"
format_stamp: "Formatted at 2026-09-27 02:33:57 on dist-test-slave-grlk"
server_key: "99ced2762e58a819cb4e4ca54fc417eb"
server_key_iv: "da3b297856eb3d7a6b7e929135cf4498"
server_key_version: "encryptionkey@0"
I20260927 02:33:57.685468 5505 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:33:57.687593 5519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:57.688722 5505 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:33:57.688825 5505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "5976a3fb096742fd81a53559e976bb22"
format_stamp: "Formatted at 2026-09-27 02:33:57 on dist-test-slave-grlk"
server_key: "99ced2762e58a819cb4e4ca54fc417eb"
server_key_iv: "da3b297856eb3d7a6b7e929135cf4498"
server_key_version: "encryptionkey@0"
I20260927 02:33:57.688906 5505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:33:57.719414 5505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:33:57.720132 5505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:33:57.720306 5505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:33:57.726740 5505 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34701
I20260927 02:33:57.726773 5571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34701 every 8 connection(s)
I20260927 02:33:57.727614 5505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:33:57.727837 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 5505
I20260927 02:33:57.727960 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:33:57.728246 392 external_mini_cluster.cc:1428] Setting key b3e4f85c04728233e164668f65ee3dc1
I20260927 02:33:57.730789 5572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:33:57.736999 5572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5976a3fb096742fd81a53559e976bb22: Bootstrap starting.
I20260927 02:33:57.737983 5575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:57.729533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48250 (local address 127.0.98.62:34701)
0927 02:33:57.730623 (+ 1090us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:57.730637 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:57.730669 (+ 32us) server_negotiation.cc:407] Connection header received
0927 02:33:57.731320 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:57.731337 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:57.731735 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:57.732028 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:57.732423 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:57.733598 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:57.734779 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:57.735403 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:57.735555 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:57.736295 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:57.736299 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:57.736311 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:33:57.736709 (+ 398us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:57.736764 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:33:57.736784 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:57.736866 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:57.737016 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 02:33:57.737280 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":764,"thread_start_us":619,"threads_started":1}
I20260927 02:33:57.739151 5572 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5976a3fb096742fd81a53559e976bb22: Neither blocks nor log segments found. Creating new log.
I20260927 02:33:57.739930 5572 log.cc:824] T 00000000000000000000000000000000 P 5976a3fb096742fd81a53559e976bb22: Log is configured to *not* fsync() on all Append() calls
I20260927 02:33:57.740572 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:33:57.742038 5572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5976a3fb096742fd81a53559e976bb22: No bootstrap required, opened a new log
I20260927 02:33:57.745378 5572 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5976a3fb096742fd81a53559e976bb22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5976a3fb096742fd81a53559e976bb22" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34701 } }
I20260927 02:33:57.745596 5572 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5976a3fb096742fd81a53559e976bb22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:33:57.745638 5572 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5976a3fb096742fd81a53559e976bb22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5976a3fb096742fd81a53559e976bb22, State: Initialized, Role: FOLLOWER
I20260927 02:33:57.746032 5572 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5976a3fb096742fd81a53559e976bb22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5976a3fb096742fd81a53559e976bb22" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34701 } }
I20260927 02:33:57.746155 5572 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5976a3fb096742fd81a53559e976bb22 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:33:57.746214 5572 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5976a3fb096742fd81a53559e976bb22 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:33:57.746305 5572 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5976a3fb096742fd81a53559e976bb22 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:33:57.747161 5572 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5976a3fb096742fd81a53559e976bb22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5976a3fb096742fd81a53559e976bb22" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34701 } }
I20260927 02:33:57.747474 5572 leader_election.cc:302] T 00000000000000000000000000000000 P 5976a3fb096742fd81a53559e976bb22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5976a3fb096742fd81a53559e976bb22; no voters:
I20260927 02:33:57.747790 5572 leader_election.cc:288] T 00000000000000000000000000000000 P 5976a3fb096742fd81a53559e976bb22 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:33:57.747874 5577 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5976a3fb096742fd81a53559e976bb22 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:33:57.748082 5577 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5976a3fb096742fd81a53559e976bb22 [term 1 LEADER]: Becoming Leader. State: Replica: 5976a3fb096742fd81a53559e976bb22, State: Running, Role: LEADER
I20260927 02:33:57.748342 5577 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5976a3fb096742fd81a53559e976bb22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5976a3fb096742fd81a53559e976bb22" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34701 } }
I20260927 02:33:57.748894 5572 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5976a3fb096742fd81a53559e976bb22 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:33:57.749810 5578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5976a3fb096742fd81a53559e976bb22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5976a3fb096742fd81a53559e976bb22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5976a3fb096742fd81a53559e976bb22" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34701 } } }
I20260927 02:33:57.749938 5578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5976a3fb096742fd81a53559e976bb22 [sys.catalog]: This master's current role is: LEADER
I20260927 02:33:57.750289 5586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:33:57.750252 5579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5976a3fb096742fd81a53559e976bb22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5976a3fb096742fd81a53559e976bb22. Latest consensus state: current_term: 1 leader_uuid: "5976a3fb096742fd81a53559e976bb22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5976a3fb096742fd81a53559e976bb22" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34701 } } }
I20260927 02:33:57.751155 5579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5976a3fb096742fd81a53559e976bb22 [sys.catalog]: This master's current role is: LEADER
I20260927 02:33:57.753307 5586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:33:57.758469 5586 catalog_manager.cc:1347] Generated new cluster ID: 2a137fa312ff4de3b4e8fcc60e29ee0b
I20260927 02:33:57.758522 5586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:33:57.787089 5586 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:33:57.788012 5586 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:33:57.795063 5586 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5976a3fb096742fd81a53559e976bb22: Generated new TSK 0
I20260927 02:33:57.795828 5586 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 02:33:57.825630 5602 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 02:33:57.825734 5602 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 02:33:57.826164 5575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:57.809980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48266 (local address 127.0.98.62:34701)
0927 02:33:57.810108 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:57.810112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:57.810297 (+ 185us) server_negotiation.cc:407] Connection header received
0927 02:33:57.810509 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:57.810512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:57.810556 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:57.810642 (+ 86us) server_negotiation.cc:226] Negotiated authn=JWT
0927 02:33:57.810870 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:57.811341 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:57.812551 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:57.812768 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:57.812836 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:57.813192 (+ 356us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 02:33:57.825882 (+ 12690us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36355/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 02:33:57.826065 (+ 183us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48266: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36355/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 02:33:57.826229 5575 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48266: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36355/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 02:33:57.827518 5601 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:33:57.809886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34701 (local address 127.0.0.1:48266)
0927 02:33:57.810217 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:57.810232 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:57.810359 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:57.810662 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:57.810665 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:57.810673 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 02:33:57.810787 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:57.810793 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:57.811448 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:57.811452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:57.812428 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:57.812438 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:57.812851 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:57.812868 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:57.812987 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:57.813035 (+ 48us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 02:33:57.826066 (+ 13031us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36355/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 02:33:57.826280 (+ 214us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:34701: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36355/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":105,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:34701: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36355/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 02:33:57.828137 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 5505
2026-09-27T02:33:57Z chronyd exiting
I20260927 02:33:57.864470 392 test_util.cc:175] -----------------------------------------------
I20260927 02:33:57.864547 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476410933508-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (323 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 02:33:57.865525 392 mini_oidc.cc:152] Starting JWKS server
I20260927 02:33:57.866319 392 webserver.cc:460] Webserver started at https://127.0.0.1:41915/ using document root <none> and password file <none>
I20260927 02:33:57.866400 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 02:33:57.866627 392 webserver.cc:460] Webserver started at http://127.0.0.1:42631/ using document root <none> and password file <none>
2026-09-27T02:33:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:33:57Z Disabled control of system clock
I20260927 02:33:57.874148 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39543
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39543
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41915/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476410933508-392-0/testchainca.cert with env {}
W20260927 02:33:57.977807 5608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:33:57.978070 5608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:33:57.978121 5608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:33:57.982136 5608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:33:57.982198 5608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:33:57.982218 5608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:33:57.982239 5608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:33:57.982259 5608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 02:33:57.982280 5608 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 02:33:57.982309 5608 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41915/jwks.json
I20260927 02:33:57.987612 5608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39543
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41915/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476410933508-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:33:57.988644 5608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:33:57.989650 5608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:33:57.997570 5616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:57.997632 5608 server_base.cc:1034] running on GCE node
W20260927 02:33:57.997582 5613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:57.997583 5614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:57.998409 5608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:33:57.999408 5608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:33:58.000548 5608 hybrid_clock.cc:648] HybridClock initialized: now 1790476438000527 us; error 36 us; skew 500 ppm
I20260927 02:33:58.002285 5608 webserver.cc:460] Webserver started at http://127.0.98.62:42455/ using document root <none> and password file <none>
I20260927 02:33:58.002810 5608 fs_manager.cc:359] Metadata directory not provided
I20260927 02:33:58.002867 5608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:33:58.003058 5608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:33:58.005074 5608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "5f79dc047e9f498db43c0a224ca659c4"
format_stamp: "Formatted at 2026-09-27 02:33:58 on dist-test-slave-grlk"
server_key: "e5cf6b0a22f78259a6f626c6f8bee2a3"
server_key_iv: "9094ef1bf43c4a921296ccbe8fa86d2a"
server_key_version: "encryptionkey@0"
I20260927 02:33:58.005653 5608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "5f79dc047e9f498db43c0a224ca659c4"
format_stamp: "Formatted at 2026-09-27 02:33:58 on dist-test-slave-grlk"
server_key: "e5cf6b0a22f78259a6f626c6f8bee2a3"
server_key_iv: "9094ef1bf43c4a921296ccbe8fa86d2a"
server_key_version: "encryptionkey@0"
I20260927 02:33:58.009318 5608 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260927 02:33:58.011487 5622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:58.012655 5608 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:33:58.012753 5608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "5f79dc047e9f498db43c0a224ca659c4"
format_stamp: "Formatted at 2026-09-27 02:33:58 on dist-test-slave-grlk"
server_key: "e5cf6b0a22f78259a6f626c6f8bee2a3"
server_key_iv: "9094ef1bf43c4a921296ccbe8fa86d2a"
server_key_version: "encryptionkey@0"
I20260927 02:33:58.012845 5608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:33:58.033625 5608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:33:58.034291 5608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:33:58.034437 5608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:33:58.040553 5608 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39543
I20260927 02:33:58.040575 5674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39543 every 8 connection(s)
I20260927 02:33:58.041450 5608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:33:58.044703 5675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:33:58.048699 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 5608
I20260927 02:33:58.048802 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:33:58.049093 392 external_mini_cluster.cc:1428] Setting key cfe5412008dda8738cdc0cecd294c889
I20260927 02:33:58.050153 5675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f79dc047e9f498db43c0a224ca659c4: Bootstrap starting.
I20260927 02:33:58.052737 5675 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f79dc047e9f498db43c0a224ca659c4: Neither blocks nor log segments found. Creating new log.
I20260927 02:33:58.053620 5675 log.cc:824] T 00000000000000000000000000000000 P 5f79dc047e9f498db43c0a224ca659c4: Log is configured to *not* fsync() on all Append() calls
I20260927 02:33:58.055466 5675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f79dc047e9f498db43c0a224ca659c4: No bootstrap required, opened a new log
I20260927 02:33:58.058840 5678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:58.050343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57576 (local address 127.0.98.62:39543)
0927 02:33:58.050972 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:58.050992 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:58.051033 (+ 41us) server_negotiation.cc:407] Connection header received
0927 02:33:58.051882 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:58.051904 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:58.052389 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:58.052738 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:58.053219 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:58.054511 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:58.055343 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:58.056006 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:58.056159 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:58.056884 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:58.056891 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:58.056907 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:33:58.057424 (+ 517us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:58.057492 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:33:58.057520 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:58.057615 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:58.057711 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 02:33:58.057964 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":109,"threads_started":1}
I20260927 02:33:58.059691 5675 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f79dc047e9f498db43c0a224ca659c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f79dc047e9f498db43c0a224ca659c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39543 } }
I20260927 02:33:58.059885 5675 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f79dc047e9f498db43c0a224ca659c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:33:58.059934 5675 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f79dc047e9f498db43c0a224ca659c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f79dc047e9f498db43c0a224ca659c4, State: Initialized, Role: FOLLOWER
I20260927 02:33:58.060308 5675 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f79dc047e9f498db43c0a224ca659c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f79dc047e9f498db43c0a224ca659c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39543 } }
I20260927 02:33:58.060444 5675 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f79dc047e9f498db43c0a224ca659c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:33:58.060511 5675 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f79dc047e9f498db43c0a224ca659c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:33:58.060585 5675 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f79dc047e9f498db43c0a224ca659c4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:33:58.061342 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:33:58.061476 5675 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f79dc047e9f498db43c0a224ca659c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f79dc047e9f498db43c0a224ca659c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39543 } }
I20260927 02:33:58.061777 5675 leader_election.cc:302] T 00000000000000000000000000000000 P 5f79dc047e9f498db43c0a224ca659c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f79dc047e9f498db43c0a224ca659c4; no voters:
I20260927 02:33:58.062036 5675 leader_election.cc:288] T 00000000000000000000000000000000 P 5f79dc047e9f498db43c0a224ca659c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:33:58.062139 5680 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f79dc047e9f498db43c0a224ca659c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:33:58.062336 5680 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f79dc047e9f498db43c0a224ca659c4 [term 1 LEADER]: Becoming Leader. State: Replica: 5f79dc047e9f498db43c0a224ca659c4, State: Running, Role: LEADER
I20260927 02:33:58.062626 5680 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f79dc047e9f498db43c0a224ca659c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f79dc047e9f498db43c0a224ca659c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39543 } }
I20260927 02:33:58.063166 5675 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f79dc047e9f498db43c0a224ca659c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:33:58.063629 5681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f79dc047e9f498db43c0a224ca659c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f79dc047e9f498db43c0a224ca659c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f79dc047e9f498db43c0a224ca659c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39543 } } }
I20260927 02:33:58.063650 5682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f79dc047e9f498db43c0a224ca659c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f79dc047e9f498db43c0a224ca659c4. Latest consensus state: current_term: 1 leader_uuid: "5f79dc047e9f498db43c0a224ca659c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f79dc047e9f498db43c0a224ca659c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39543 } } }
I20260927 02:33:58.063768 5681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f79dc047e9f498db43c0a224ca659c4 [sys.catalog]: This master's current role is: LEADER
I20260927 02:33:58.063768 5682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f79dc047e9f498db43c0a224ca659c4 [sys.catalog]: This master's current role is: LEADER
I20260927 02:33:58.064148 5685 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:33:58.066962 5685 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:33:58.072065 5685 catalog_manager.cc:1347] Generated new cluster ID: ce8e6b82316343cb8d43a02893f9ce49
I20260927 02:33:58.072121 5685 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:33:58.098963 5685 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:33:58.099823 5685 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:33:58.119750 5685 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f79dc047e9f498db43c0a224ca659c4: Generated new TSK 0
I20260927 02:33:58.120394 5685 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 02:33:58.145470 5705 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 02:33:58.145546 5705 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 02:33:58.146234 5678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:58.130814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57578 (local address 127.0.98.62:39543)
0927 02:33:58.130935 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:58.130940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:58.131025 (+ 85us) server_negotiation.cc:407] Connection header received
0927 02:33:58.131163 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:58.131166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:58.131220 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:58.131338 (+ 118us) server_negotiation.cc:226] Negotiated authn=JWT
0927 02:33:58.131492 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:58.132259 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:58.133110 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:58.133472 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:58.133589 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:58.133821 (+ 232us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 02:33:58.145845 (+ 12024us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41915/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 02:33:58.146093 (+ 248us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57578: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41915/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 02:33:58.146299 5678 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57578: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41915/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260927 02:33:58.146353 5704 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:33:58.130714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39543 (local address 127.0.0.1:57578)
0927 02:33:58.130944 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:58.130956 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:58.131044 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:58.131326 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:58.131328 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:58.131333 (+ 5us) client_negotiation.cc:189] Negotiated authn=JWT
0927 02:33:58.131417 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:58.131420 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:58.132368 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:58.132370 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:58.133006 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:58.133013 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:58.133570 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:58.133584 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:58.133645 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:58.133666 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 02:33:58.146052 (+ 12386us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41915/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 02:33:58.146202 (+ 150us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:39543: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41915/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":78,"threads_started":1}
I20260927 02:33:58.146947 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 5608
2026-09-27T02:33:58Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (319 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 02:33:58.184899 392 mini_oidc.cc:152] Starting JWKS server
I20260927 02:33:58.185761 392 webserver.cc:460] Webserver started at https://127.0.0.1:39067/ using document root <none> and password file <none>
I20260927 02:33:58.185843 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 02:33:58.186039 392 webserver.cc:460] Webserver started at http://127.0.0.1:38195/ using document root <none> and password file <none>
2026-09-27T02:33:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:33:58Z Disabled control of system clock
I20260927 02:33:58.193962 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36475
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36475
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39067/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 02:33:58.298830 5711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:33:58.299062 5711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:33:58.299098 5711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:33:58.302989 5711 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:33:58.303045 5711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:33:58.303059 5711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:33:58.303074 5711 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:33:58.303087 5711 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 02:33:58.303107 5711 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 02:33:58.303133 5711 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39067/jwks.json
I20260927 02:33:58.307642 5711 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36475
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39067/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:33:58.308593 5711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:33:58.309876 5711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:33:58.317411 5716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:58.317417 5719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:58.317417 5717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:58.317653 5711 server_base.cc:1034] running on GCE node
I20260927 02:33:58.318120 5711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:33:58.319104 5711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:33:58.320255 5711 hybrid_clock.cc:648] HybridClock initialized: now 1790476438320242 us; error 28 us; skew 500 ppm
I20260927 02:33:58.321911 5711 webserver.cc:460] Webserver started at http://127.0.98.62:45265/ using document root <none> and password file <none>
I20260927 02:33:58.322412 5711 fs_manager.cc:359] Metadata directory not provided
I20260927 02:33:58.322474 5711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:33:58.322652 5711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:33:58.324509 5711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "b53269da367146228b805e3e5a7673d2"
format_stamp: "Formatted at 2026-09-27 02:33:58 on dist-test-slave-grlk"
server_key: "1a72167b905d1671180e1db85943b276"
server_key_iv: "106ac301c726e6e83cb6885b381f5621"
server_key_version: "encryptionkey@0"
I20260927 02:33:58.325054 5711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "b53269da367146228b805e3e5a7673d2"
format_stamp: "Formatted at 2026-09-27 02:33:58 on dist-test-slave-grlk"
server_key: "1a72167b905d1671180e1db85943b276"
server_key_iv: "106ac301c726e6e83cb6885b381f5621"
server_key_version: "encryptionkey@0"
I20260927 02:33:58.328344 5711 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 02:33:58.330700 5725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:58.331856 5711 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:33:58.331961 5711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "b53269da367146228b805e3e5a7673d2"
format_stamp: "Formatted at 2026-09-27 02:33:58 on dist-test-slave-grlk"
server_key: "1a72167b905d1671180e1db85943b276"
server_key_iv: "106ac301c726e6e83cb6885b381f5621"
server_key_version: "encryptionkey@0"
I20260927 02:33:58.332072 5711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:33:58.350618 5711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:33:58.351214 5711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:33:58.351366 5711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:33:58.357465 5711 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36475
I20260927 02:33:58.357476 5777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36475 every 8 connection(s)
I20260927 02:33:58.358299 5711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:33:58.358647 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 5711
I20260927 02:33:58.358747 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:33:58.359020 392 external_mini_cluster.cc:1428] Setting key 30583c51ba773c5b322437927369985c
I20260927 02:33:58.361471 5778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:33:58.367252 5778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b53269da367146228b805e3e5a7673d2: Bootstrap starting.
I20260927 02:33:58.367477 5781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:58.360258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45740 (local address 127.0.98.62:36475)
0927 02:33:58.360831 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:58.360845 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:58.360878 (+ 33us) server_negotiation.cc:407] Connection header received
0927 02:33:58.361530 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:58.361546 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:58.361941 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:58.362206 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:58.362514 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:58.363435 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:58.364585 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:58.365060 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:58.365165 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:58.365931 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:58.365935 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:58.365946 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:33:58.366286 (+ 340us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:58.366334 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:33:58.366351 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:58.366419 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:58.366569 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 02:33:58.366815 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":109,"threads_started":1}
I20260927 02:33:58.369374 5778 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b53269da367146228b805e3e5a7673d2: Neither blocks nor log segments found. Creating new log.
I20260927 02:33:58.369839 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:33:58.370121 5778 log.cc:824] T 00000000000000000000000000000000 P b53269da367146228b805e3e5a7673d2: Log is configured to *not* fsync() on all Append() calls
I20260927 02:33:58.372030 5778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b53269da367146228b805e3e5a7673d2: No bootstrap required, opened a new log
I20260927 02:33:58.375495 5778 raft_consensus.cc:348] T 00000000000000000000000000000000 P b53269da367146228b805e3e5a7673d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53269da367146228b805e3e5a7673d2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36475 } }
I20260927 02:33:58.375700 5778 raft_consensus.cc:374] T 00000000000000000000000000000000 P b53269da367146228b805e3e5a7673d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:33:58.375739 5778 raft_consensus.cc:729] T 00000000000000000000000000000000 P b53269da367146228b805e3e5a7673d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b53269da367146228b805e3e5a7673d2, State: Initialized, Role: FOLLOWER
I20260927 02:33:58.376084 5778 consensus_queue.cc:260] T 00000000000000000000000000000000 P b53269da367146228b805e3e5a7673d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53269da367146228b805e3e5a7673d2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36475 } }
I20260927 02:33:58.376184 5778 raft_consensus.cc:388] T 00000000000000000000000000000000 P b53269da367146228b805e3e5a7673d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:33:58.376225 5778 raft_consensus.cc:482] T 00000000000000000000000000000000 P b53269da367146228b805e3e5a7673d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:33:58.376289 5778 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b53269da367146228b805e3e5a7673d2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:33:58.377090 5778 raft_consensus.cc:504] T 00000000000000000000000000000000 P b53269da367146228b805e3e5a7673d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53269da367146228b805e3e5a7673d2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36475 } }
I20260927 02:33:58.377364 5778 leader_election.cc:302] T 00000000000000000000000000000000 P b53269da367146228b805e3e5a7673d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b53269da367146228b805e3e5a7673d2; no voters:
I20260927 02:33:58.377597 5778 leader_election.cc:288] T 00000000000000000000000000000000 P b53269da367146228b805e3e5a7673d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:33:58.377689 5783 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b53269da367146228b805e3e5a7673d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:33:58.377873 5783 raft_consensus.cc:686] T 00000000000000000000000000000000 P b53269da367146228b805e3e5a7673d2 [term 1 LEADER]: Becoming Leader. State: Replica: b53269da367146228b805e3e5a7673d2, State: Running, Role: LEADER
I20260927 02:33:58.378167 5783 consensus_queue.cc:237] T 00000000000000000000000000000000 P b53269da367146228b805e3e5a7673d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53269da367146228b805e3e5a7673d2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36475 } }
I20260927 02:33:58.378636 5778 sys_catalog.cc:564] T 00000000000000000000000000000000 P b53269da367146228b805e3e5a7673d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:33:58.379572 5785 sys_catalog.cc:455] T 00000000000000000000000000000000 P b53269da367146228b805e3e5a7673d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b53269da367146228b805e3e5a7673d2. Latest consensus state: current_term: 1 leader_uuid: "b53269da367146228b805e3e5a7673d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53269da367146228b805e3e5a7673d2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36475 } } }
I20260927 02:33:58.379688 5785 sys_catalog.cc:458] T 00000000000000000000000000000000 P b53269da367146228b805e3e5a7673d2 [sys.catalog]: This master's current role is: LEADER
I20260927 02:33:58.379935 5784 sys_catalog.cc:455] T 00000000000000000000000000000000 P b53269da367146228b805e3e5a7673d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b53269da367146228b805e3e5a7673d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53269da367146228b805e3e5a7673d2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36475 } } }
I20260927 02:33:58.380019 5784 sys_catalog.cc:458] T 00000000000000000000000000000000 P b53269da367146228b805e3e5a7673d2 [sys.catalog]: This master's current role is: LEADER
I20260927 02:33:58.380085 5792 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:33:58.382979 5792 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:33:58.388054 5792 catalog_manager.cc:1347] Generated new cluster ID: 953793aad2014e188b3dfe226ba7a58a
I20260927 02:33:58.388113 5792 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:33:58.397910 5792 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:33:58.398965 5792 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:33:58.405311 5792 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b53269da367146228b805e3e5a7673d2: Generated new TSK 0
I20260927 02:33:58.405982 5792 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:33:58.424706 5781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:58.419807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45756 (local address 127.0.98.62:36475)
0927 02:33:58.419933 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:58.419938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:58.420111 (+ 173us) server_negotiation.cc:407] Connection header received
0927 02:33:58.420257 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:58.420260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:58.420305 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:58.420392 (+ 87us) server_negotiation.cc:226] Negotiated authn=JWT
0927 02:33:58.420609 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:58.421169 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:58.422355 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:58.422564 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:58.422636 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:58.422922 (+ 286us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 02:33:58.424319 (+ 1397us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39067/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 02:33:58.424559 (+ 240us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45756: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39067/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 02:33:58.424788 5807 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:33:58.419718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36475 (local address 127.0.0.1:45756)
0927 02:33:58.420014 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:58.420028 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:58.420149 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:58.420403 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:58.420406 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:58.420413 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0927 02:33:58.420523 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:58.420529 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:58.421267 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:58.421271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:58.422253 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:58.422263 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:58.422648 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:58.422670 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:58.422756 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:58.422784 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 02:33:58.424472 (+ 1688us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39067/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 02:33:58.424613 (+ 141us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:36475: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39067/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":100,"threads_started":1}
W20260927 02:33:58.424793 5781 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45756: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39067/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260927 02:33:58.425365 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 5711
E20260927 02:33:58.435705 5808 webserver.cc:533] Webserver: sslize failed:
E20260927 02:33:58.435765 5808 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-27T02:33:58Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (278 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T02:33:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:33:58Z Disabled control of system clock
I20260927 02:33:58.470556 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35813
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35813
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476410933508-392-0/testchainca.cert with env {}
W20260927 02:33:58.571543 5812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:33:58.571768 5812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:33:58.571803 5812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:33:58.575711 5812 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:33:58.575765 5812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:33:58.575779 5812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:33:58.575796 5812 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:33:58.575824 5812 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 02:33:58.575851 5812 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 02:33:58.575875 5812 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 02:33:58.580968 5812 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35813
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476410933508-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:33:58.581908 5812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:33:58.582842 5812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:33:58.590407 5817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:58.590533 5812 server_base.cc:1034] running on GCE node
W20260927 02:33:58.590443 5820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:33:58.590467 5818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:33:58.591190 5812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:33:58.592151 5812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:33:58.593292 5812 hybrid_clock.cc:648] HybridClock initialized: now 1790476438593273 us; error 31 us; skew 500 ppm
I20260927 02:33:58.594939 5812 webserver.cc:460] Webserver started at http://127.0.98.62:40093/ using document root <none> and password file <none>
I20260927 02:33:58.595431 5812 fs_manager.cc:359] Metadata directory not provided
I20260927 02:33:58.595495 5812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:33:58.595665 5812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:33:58.597532 5812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "a9dc34fd1a3248b39eedf5e317b7208d"
format_stamp: "Formatted at 2026-09-27 02:33:58 on dist-test-slave-grlk"
server_key: "866a821bf4a5570cb10a146f86773e88"
server_key_iv: "6ef344993fdffab02d5765bfea28a946"
server_key_version: "encryptionkey@0"
I20260927 02:33:58.598021 5812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "a9dc34fd1a3248b39eedf5e317b7208d"
format_stamp: "Formatted at 2026-09-27 02:33:58 on dist-test-slave-grlk"
server_key: "866a821bf4a5570cb10a146f86773e88"
server_key_iv: "6ef344993fdffab02d5765bfea28a946"
server_key_version: "encryptionkey@0"
I20260927 02:33:58.601239 5812 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:33:58.603312 5826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:33:58.604357 5812 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:33:58.604467 5812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "a9dc34fd1a3248b39eedf5e317b7208d"
format_stamp: "Formatted at 2026-09-27 02:33:58 on dist-test-slave-grlk"
server_key: "866a821bf4a5570cb10a146f86773e88"
server_key_iv: "6ef344993fdffab02d5765bfea28a946"
server_key_version: "encryptionkey@0"
I20260927 02:33:58.604549 5812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:33:58.617538 5812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:33:58.618103 5812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:33:58.618249 5812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:33:58.624222 5812 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35813
I20260927 02:33:58.624234 5878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35813 every 8 connection(s)
I20260927 02:33:58.625094 5812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:33:58.627589 5879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:33:58.632401 5879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9dc34fd1a3248b39eedf5e317b7208d: Bootstrap starting.
I20260927 02:33:58.634593 5879 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9dc34fd1a3248b39eedf5e317b7208d: Neither blocks nor log segments found. Creating new log.
I20260927 02:33:58.634907 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 5812
I20260927 02:33:58.635020 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:33:58.635278 392 external_mini_cluster.cc:1428] Setting key ac40a831de8f7d269b203e45ac5d14a2
I20260927 02:33:58.635396 5879 log.cc:824] T 00000000000000000000000000000000 P a9dc34fd1a3248b39eedf5e317b7208d: Log is configured to *not* fsync() on all Append() calls
I20260927 02:33:58.637460 5879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9dc34fd1a3248b39eedf5e317b7208d: No bootstrap required, opened a new log
I20260927 02:33:58.641589 5879 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9dc34fd1a3248b39eedf5e317b7208d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9dc34fd1a3248b39eedf5e317b7208d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35813 } }
I20260927 02:33:58.641840 5879 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9dc34fd1a3248b39eedf5e317b7208d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:33:58.641903 5879 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9dc34fd1a3248b39eedf5e317b7208d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9dc34fd1a3248b39eedf5e317b7208d, State: Initialized, Role: FOLLOWER
I20260927 02:33:58.642351 5879 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9dc34fd1a3248b39eedf5e317b7208d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9dc34fd1a3248b39eedf5e317b7208d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35813 } }
I20260927 02:33:58.642482 5879 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9dc34fd1a3248b39eedf5e317b7208d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:33:58.642545 5879 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9dc34fd1a3248b39eedf5e317b7208d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:33:58.642640 5879 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9dc34fd1a3248b39eedf5e317b7208d [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:33:58.643450 5879 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9dc34fd1a3248b39eedf5e317b7208d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9dc34fd1a3248b39eedf5e317b7208d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35813 } }
I20260927 02:33:58.643756 5879 leader_election.cc:302] T 00000000000000000000000000000000 P a9dc34fd1a3248b39eedf5e317b7208d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9dc34fd1a3248b39eedf5e317b7208d; no voters:
I20260927 02:33:58.644017 5879 leader_election.cc:288] T 00000000000000000000000000000000 P a9dc34fd1a3248b39eedf5e317b7208d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:33:58.644099 5884 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9dc34fd1a3248b39eedf5e317b7208d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:33:58.644269 5884 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9dc34fd1a3248b39eedf5e317b7208d [term 1 LEADER]: Becoming Leader. State: Replica: a9dc34fd1a3248b39eedf5e317b7208d, State: Running, Role: LEADER
I20260927 02:33:58.644551 5884 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9dc34fd1a3248b39eedf5e317b7208d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9dc34fd1a3248b39eedf5e317b7208d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35813 } }
I20260927 02:33:58.644596 5883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:58.636238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44908 (local address 127.0.98.62:35813)
0927 02:33:58.636861 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:58.636881 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:58.636922 (+ 41us) server_negotiation.cc:407] Connection header received
0927 02:33:58.637741 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:58.637765 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:58.638153 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:58.638492 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:33:58.638914 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:58.640346 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:58.641246 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:58.641914 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:58.642072 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:58.642817 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:33:58.642822 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:33:58.642834 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:33:58.643188 (+ 354us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:33:58.643241 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:33:58.643260 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:33:58.643341 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 02:33:58.643473 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 02:33:58.643754 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":92,"threads_started":1}
I20260927 02:33:58.645236 5879 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9dc34fd1a3248b39eedf5e317b7208d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:33:58.645712 5886 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9dc34fd1a3248b39eedf5e317b7208d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9dc34fd1a3248b39eedf5e317b7208d. Latest consensus state: current_term: 1 leader_uuid: "a9dc34fd1a3248b39eedf5e317b7208d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9dc34fd1a3248b39eedf5e317b7208d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35813 } } }
I20260927 02:33:58.645825 5886 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9dc34fd1a3248b39eedf5e317b7208d [sys.catalog]: This master's current role is: LEADER
I20260927 02:33:58.646193 5890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:33:58.646423 5885 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9dc34fd1a3248b39eedf5e317b7208d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9dc34fd1a3248b39eedf5e317b7208d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9dc34fd1a3248b39eedf5e317b7208d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35813 } } }
I20260927 02:33:58.646516 5885 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9dc34fd1a3248b39eedf5e317b7208d [sys.catalog]: This master's current role is: LEADER
I20260927 02:33:58.647541 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:33:58.649344 5890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:33:58.654404 5890 catalog_manager.cc:1347] Generated new cluster ID: 58f78488fed845e5a351d2846c7140db
I20260927 02:33:58.654455 5890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:33:58.664714 5890 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:33:58.665618 5890 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:33:58.670845 5890 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9dc34fd1a3248b39eedf5e317b7208d: Generated new TSK 0
I20260927 02:33:58.671419 5890 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:33:58.710435 5883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:33:58.680001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44922 (local address 127.0.98.62:35813)
0927 02:33:58.680112 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 02:33:58.680116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:33:58.680285 (+ 169us) server_negotiation.cc:407] Connection header received
0927 02:33:58.680437 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:33:58.680439 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:33:58.680483 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:33:58.680567 (+ 84us) server_negotiation.cc:226] Negotiated authn=JWT
0927 02:33:58.680804 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:58.681364 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:58.682194 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:33:58.682407 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:33:58.682489 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:58.682764 (+ 275us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 02:33:58.709972 (+ 27208us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 02:33:58.710288 (+ 316us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44922: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":39}
W20260927 02:33:58.710527 5883 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44922: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 02:33:58.710552 5908 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:33:58.679908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35813 (local address 127.0.0.1:44922)
0927 02:33:58.680195 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 02:33:58.680207 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:33:58.680323 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:33:58.680586 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:33:58.680589 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:33:58.680597 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 02:33:58.680713 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:58.680718 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:58.681463 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:58.681467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:58.682087 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:33:58.682094 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:33:58.682495 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:33:58.682514 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:33:58.682596 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:33:58.682619 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 02:33:58.710200 (+ 27581us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 02:33:58.710381 (+ 181us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:35813: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":95,"threads_started":1}
I20260927 02:33:58.711283 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 5812
2026-09-27T02:33:58Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (265 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (114 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (114 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:33:58 dist-test-slave-grlk krb5kdc[5920](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:33:58 dist-test-slave-grlk krb5kdc[5920](info): set up 2 sockets
Sep 27 02:33:58 dist-test-slave-grlk krb5kdc[5920](info): commencing operation
krb5kdc: starting...
W20260927 02:33:59.892835 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/krb5kdc/test-user.keytab.
Sep 27 02:33:59 dist-test-slave-grlk krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:33:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:33:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:34:00.009090 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42979
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42979
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:34:00.111738 5933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:34:00.111968 5933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:34:00.112005 5933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:34:00.115911 5933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:34:00.115962 5933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:34:00.115980 5933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:34:00.115999 5933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:34:00.116016 5933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:34:00.120959 5933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42979
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:34:00.121824 5933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:34:00.122696 5933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:34:00.130311 5938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:00.130353 5933 server_base.cc:1034] running on GCE node
W20260927 02:34:00.130475 5939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:00.130357 5941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:00.131052 5933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:34:00.132046 5933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:34:00.133195 5933 hybrid_clock.cc:648] HybridClock initialized: now 1790476440133181 us; error 26 us; skew 500 ppm
Sep 27 02:34:00 dist-test-slave-grlk krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:34:00.135654 5933 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:34:00.136420 5933 webserver.cc:460] Webserver started at http://127.0.98.62:46433/ using document root <none> and password file <none>
I20260927 02:34:00.136914 5933 fs_manager.cc:359] Metadata directory not provided
I20260927 02:34:00.137017 5933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:34:00.137212 5933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:34:00.139086 5933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "0dd0ae5163f14317bd872a8913a45800"
format_stamp: "Formatted at 2026-09-27 02:34:00 on dist-test-slave-grlk"
server_key: "fa1aad1b0500888493a377cfd1ce1a11"
server_key_iv: "5d15f17ffd21be316beddec898db89f9"
server_key_version: "encryptionkey@0"
I20260927 02:34:00.139568 5933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "0dd0ae5163f14317bd872a8913a45800"
format_stamp: "Formatted at 2026-09-27 02:34:00 on dist-test-slave-grlk"
server_key: "fa1aad1b0500888493a377cfd1ce1a11"
server_key_iv: "5d15f17ffd21be316beddec898db89f9"
server_key_version: "encryptionkey@0"
I20260927 02:34:00.142889 5933 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:34:00.145006 5948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:00.146095 5933 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:34:00.146209 5933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "0dd0ae5163f14317bd872a8913a45800"
format_stamp: "Formatted at 2026-09-27 02:34:00 on dist-test-slave-grlk"
server_key: "fa1aad1b0500888493a377cfd1ce1a11"
server_key_iv: "5d15f17ffd21be316beddec898db89f9"
server_key_version: "encryptionkey@0"
I20260927 02:34:00.146292 5933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:34:00.179517 5933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:34:00.182232 5933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:34:00.182401 5933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:34:00.188453 5933 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42979
I20260927 02:34:00.188694 6000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42979 every 8 connection(s)
I20260927 02:34:00.189330 5933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:34:00.191824 6001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:34:00.193816 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 5933
I20260927 02:34:00.193904 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:34:00.194159 392 external_mini_cluster.cc:1428] Setting key d03087312f2aa2aeb9895de5fbe4303b
I20260927 02:34:00.197227 6001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dd0ae5163f14317bd872a8913a45800: Bootstrap starting.
I20260927 02:34:00.199479 6001 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0dd0ae5163f14317bd872a8913a45800: Neither blocks nor log segments found. Creating new log.
I20260927 02:34:00.200191 6001 log.cc:824] T 00000000000000000000000000000000 P 0dd0ae5163f14317bd872a8913a45800: Log is configured to *not* fsync() on all Append() calls
I20260927 02:34:00.202178 6001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dd0ae5163f14317bd872a8913a45800: No bootstrap required, opened a new log
Sep 27 02:34:00 dist-test-slave-grlk krb5kdc[5920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:34:00.205575 6001 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0dd0ae5163f14317bd872a8913a45800 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dd0ae5163f14317bd872a8913a45800" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42979 } }
I20260927 02:34:00.205793 6001 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0dd0ae5163f14317bd872a8913a45800 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:34:00.205861 6001 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0dd0ae5163f14317bd872a8913a45800 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dd0ae5163f14317bd872a8913a45800, State: Initialized, Role: FOLLOWER
I20260927 02:34:00.206274 6001 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0dd0ae5163f14317bd872a8913a45800 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dd0ae5163f14317bd872a8913a45800" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42979 } }
I20260927 02:34:00.206401 6001 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0dd0ae5163f14317bd872a8913a45800 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:34:00.206462 6001 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0dd0ae5163f14317bd872a8913a45800 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:34:00.206552 6001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0dd0ae5163f14317bd872a8913a45800 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:34:00.206840 6004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:00.195393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51344 (local address 127.0.98.62:42979)
0927 02:34:00.195733 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:00.195746 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:00.195778 (+ 32us) server_negotiation.cc:407] Connection header received
0927 02:34:00.196378 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:00.196393 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:00.196694 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:00.196970 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:00.197775 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:00.198683 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:00.199755 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:00.200181 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:00.200298 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:00.202778 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:00.202783 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:00.202793 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:00.202816 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:00.204244 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:00.204610 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:00.204616 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:00.204620 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:00.204725 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:00.204983 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:00.204986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:00.204988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:00.205335 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:00.205517 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:00.205781 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 02:34:00.206055 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":85,"threads_started":1}
I20260927 02:34:00.207427 6001 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0dd0ae5163f14317bd872a8913a45800 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dd0ae5163f14317bd872a8913a45800" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42979 } }
I20260927 02:34:00.207728 6001 leader_election.cc:302] T 00000000000000000000000000000000 P 0dd0ae5163f14317bd872a8913a45800 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0dd0ae5163f14317bd872a8913a45800; no voters:
I20260927 02:34:00.207978 6001 leader_election.cc:288] T 00000000000000000000000000000000 P 0dd0ae5163f14317bd872a8913a45800 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:34:00.208087 6006 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0dd0ae5163f14317bd872a8913a45800 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:34:00.208276 6006 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0dd0ae5163f14317bd872a8913a45800 [term 1 LEADER]: Becoming Leader. State: Replica: 0dd0ae5163f14317bd872a8913a45800, State: Running, Role: LEADER
I20260927 02:34:00.208595 6006 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0dd0ae5163f14317bd872a8913a45800 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dd0ae5163f14317bd872a8913a45800" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42979 } }
I20260927 02:34:00.209055 6001 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0dd0ae5163f14317bd872a8913a45800 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:34:00.209633 6006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dd0ae5163f14317bd872a8913a45800 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0dd0ae5163f14317bd872a8913a45800. Latest consensus state: current_term: 1 leader_uuid: "0dd0ae5163f14317bd872a8913a45800" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dd0ae5163f14317bd872a8913a45800" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42979 } } }
I20260927 02:34:00.209800 6006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dd0ae5163f14317bd872a8913a45800 [sys.catalog]: This master's current role is: LEADER
I20260927 02:34:00.209995 6007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dd0ae5163f14317bd872a8913a45800 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0dd0ae5163f14317bd872a8913a45800" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dd0ae5163f14317bd872a8913a45800" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42979 } } }
I20260927 02:34:00.210103 6007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dd0ae5163f14317bd872a8913a45800 [sys.catalog]: This master's current role is: LEADER
I20260927 02:34:00.210265 6015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:34:00.212872 6015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:34:00.217824 6015 catalog_manager.cc:1347] Generated new cluster ID: 81db2610292947ff81bbf839e15161b6
I20260927 02:34:00.217873 6015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:34:00.226807 6015 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:34:00.227567 6015 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:34:00.234163 6015 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0dd0ae5163f14317bd872a8913a45800: Generated new TSK 0
I20260927 02:34:00.234685 6015 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:34:00.264038 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42979
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:34:00.375110 6027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:34:00.375344 6027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:34:00.375382 6027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:34:00.379284 6027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:34:00.379330 6027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:34:00.379383 6027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:34:00.384243 6027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:34:00.385139 6027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:34:00.386041 6027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:34:00.394600 6033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:00.394745 6027 server_base.cc:1034] running on GCE node
W20260927 02:34:00.394636 6035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:00.394621 6032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:00.395164 6027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:34:00.395704 6027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:34:00.396847 6027 hybrid_clock.cc:648] HybridClock initialized: now 1790476440396831 us; error 27 us; skew 500 ppm
Sep 27 02:34:00 dist-test-slave-grlk krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:34:00.399246 6027 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:34:00.399977 6027 webserver.cc:460] Webserver started at http://127.0.98.1:40599/ using document root <none> and password file <none>
I20260927 02:34:00.400440 6027 fs_manager.cc:359] Metadata directory not provided
I20260927 02:34:00.400494 6027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:34:00.400663 6027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:34:00.402505 6027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/ts-0/data/instance:
uuid: "37d33c17138549f7aa8ca31aa661af48"
format_stamp: "Formatted at 2026-09-27 02:34:00 on dist-test-slave-grlk"
server_key: "671d6c3614319db91c99459f80942001"
server_key_iv: "af294439d7dea77c5fe4b91e91d9e13f"
server_key_version: "encryptionkey@0"
I20260927 02:34:00.402997 6027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/ts-0/wal/instance:
uuid: "37d33c17138549f7aa8ca31aa661af48"
format_stamp: "Formatted at 2026-09-27 02:34:00 on dist-test-slave-grlk"
server_key: "671d6c3614319db91c99459f80942001"
server_key_iv: "af294439d7dea77c5fe4b91e91d9e13f"
server_key_version: "encryptionkey@0"
I20260927 02:34:00.406185 6027 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:34:00.408209 6042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:00.409338 6027 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:34:00.409428 6027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/ts-0/wal
uuid: "37d33c17138549f7aa8ca31aa661af48"
format_stamp: "Formatted at 2026-09-27 02:34:00 on dist-test-slave-grlk"
server_key: "671d6c3614319db91c99459f80942001"
server_key_iv: "af294439d7dea77c5fe4b91e91d9e13f"
server_key_version: "encryptionkey@0"
I20260927 02:34:00.409507 6027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:34:00.420701 6027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:34:00.423311 6027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:34:00.423468 6027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:34:00.423982 6027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:34:00.424737 6027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:34:00.424782 6027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:00.424831 6027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:34:00.424858 6027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:00.432730 6027 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39079
I20260927 02:34:00.432763 6155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39079 every 8 connection(s)
I20260927 02:34:00.433561 6027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:34:00.439532 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 6027
I20260927 02:34:00.439656 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:34:00.439916 392 external_mini_cluster.cc:1428] Setting key 4d37461c3e1bb79336b36fb5aabe0a2b
Sep 27 02:34:00 dist-test-slave-grlk krb5kdc[5920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:34:00.446354 6004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:00.435191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52497 (local address 127.0.98.62:42979)
0927 02:34:00.435304 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:00.435307 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:00.436051 (+ 744us) server_negotiation.cc:407] Connection header received
0927 02:34:00.437353 (+ 1302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:00.437356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:00.437401 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:00.437472 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:00.438473 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:00.438983 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:00.440358 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:00.440613 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:00.440709 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:00.443124 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:00.443128 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:00.443130 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:00.443153 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:00.444405 (+ 1252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:00.444952 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:00.444956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:00.444958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:00.445047 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:00.445381 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:00.445383 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:00.445384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:00.445539 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:00.445609 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:00.446149 (+ 540us) server_negotiation.cc:299] Negotiation successful
0927 02:34:00.446243 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 02:34:00.447381 6158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:00.435508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42979 (local address 127.0.98.1:52497)
0927 02:34:00.435912 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:00.435949 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:00.437170 (+ 1221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:00.437617 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:00.437624 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:00.437934 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:00.438369 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:00.438381 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:00.439064 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:00.439067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:00.440257 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:00.440268 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:00.440745 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:00.440748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:00.440885 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:00.441457 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:00.441475 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:00.442981 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:34:00.444518 (+ 1537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:00.444523 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:00.444536 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:00.444850 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:00.445129 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:00.445132 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:00.445134 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:00.445302 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:00.445628 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:34:00.445634 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:34:00.445914 (+ 280us) client_negotiation.cc:769] Sending connection context
0927 02:34:00.446130 (+ 216us) client_negotiation.cc:240] Negotiation successful
0927 02:34:00.446433 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":102,"threads_started":1}
I20260927 02:34:00.448415 6156 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42979
I20260927 02:34:00.448630 6156 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:34:00.449165 6156 heartbeater.cc:511] Master 127.0.98.62:42979 requested a full tablet report, sending...
I20260927 02:34:00.450484 5965 ts_manager.cc:194] Registered new tserver with Master: 37d33c17138549f7aa8ca31aa661af48 (127.0.98.1:39079)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 02:34:00.451624 5965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52497
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:34:00.463421 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42979
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:34:00.565604 6161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:34:00.565829 6161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:34:00.565886 6161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:34:00.569766 6161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:34:00.569818 6161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:34:00.569895 6161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:34:00.574822 6161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:34:00.575729 6161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:34:00.576635 6161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:34:00.585243 6169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:00.585316 6166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:00.585392 6161 server_base.cc:1034] running on GCE node
W20260927 02:34:00.585247 6167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:00.585796 6161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:34:00.586326 6161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:34:00.587471 6161 hybrid_clock.cc:648] HybridClock initialized: now 1790476440587454 us; error 26 us; skew 500 ppm
Sep 27 02:34:00 dist-test-slave-grlk krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:34:00.589885 6161 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:34:00.590617 6161 webserver.cc:460] Webserver started at http://127.0.98.2:38165/ using document root <none> and password file <none>
I20260927 02:34:00.591075 6161 fs_manager.cc:359] Metadata directory not provided
I20260927 02:34:00.591136 6161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:34:00.591327 6161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:34:00.593184 6161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/ts-1/data/instance:
uuid: "5b6d66fca4c34374a324e742db9e1a82"
format_stamp: "Formatted at 2026-09-27 02:34:00 on dist-test-slave-grlk"
server_key: "ce4e3ceae19dbf62c71640711ebb4e11"
server_key_iv: "a0df5880e36bef709416fed04bfe4463"
server_key_version: "encryptionkey@0"
I20260927 02:34:00.593689 6161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5b6d66fca4c34374a324e742db9e1a82"
format_stamp: "Formatted at 2026-09-27 02:34:00 on dist-test-slave-grlk"
server_key: "ce4e3ceae19dbf62c71640711ebb4e11"
server_key_iv: "a0df5880e36bef709416fed04bfe4463"
server_key_version: "encryptionkey@0"
I20260927 02:34:00.597087 6161 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:34:00.599375 6176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:00.600522 6161 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:34:00.600631 6161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/ts-1/wal
uuid: "5b6d66fca4c34374a324e742db9e1a82"
format_stamp: "Formatted at 2026-09-27 02:34:00 on dist-test-slave-grlk"
server_key: "ce4e3ceae19dbf62c71640711ebb4e11"
server_key_iv: "a0df5880e36bef709416fed04bfe4463"
server_key_version: "encryptionkey@0"
I20260927 02:34:00.600726 6161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:34:00.623629 6161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:34:00.626677 6161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:34:00.626849 6161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:34:00.627395 6161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:34:00.628213 6161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:34:00.628264 6161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:00.628319 6161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:34:00.628347 6161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:00.636869 6161 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42355
I20260927 02:34:00.636922 6289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42355 every 8 connection(s)
I20260927 02:34:00.637763 6161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:34:00.638120 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 6161
I20260927 02:34:00.638238 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:34:00.638636 392 external_mini_cluster.cc:1428] Setting key e46416c0cbb79548ed3c6a5b3491643b
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Sep 27 02:34:00 dist-test-slave-grlk krb5kdc[5920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 02:34:00.650513 6004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:00.639431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59179 (local address 127.0.98.62:42979)
0927 02:34:00.639555 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:00.639559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:00.640422 (+ 863us) server_negotiation.cc:407] Connection header received
0927 02:34:00.641729 (+ 1307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:00.641732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:00.641782 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:00.641851 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:00.642890 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:00.643462 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:00.644524 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:00.644804 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:00.644905 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:00.647164 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:00.647168 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:00.647170 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:00.647192 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:00.648476 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:00.649024 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:00.649027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:00.649029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:00.649111 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:00.649465 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:00.649469 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:00.649470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:00.649678 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:00.649763 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:00.650223 (+ 460us) server_negotiation.cc:299] Negotiation successful
0927 02:34:00.650337 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 02:34:00.651201 6292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:00.639846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42979 (local address 127.0.98.2:59179)
0927 02:34:00.640289 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:00.640329 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:00.641532 (+ 1203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:00.642000 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:00.642007 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:00.642329 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:00.642762 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:00.642774 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:00.643569 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:00.643571 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:00.644416 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:00.644425 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:00.644958 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:00.644963 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:00.645111 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:00.645602 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:00.645617 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:00.647020 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:34:00.648594 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:00.648600 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:00.648612 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:00.648905 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:00.649205 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:00.649208 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:00.649209 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:00.649369 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:00.649770 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:34:00.649774 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:34:00.649976 (+ 202us) client_negotiation.cc:769] Sending connection context
0927 02:34:00.650158 (+ 182us) client_negotiation.cc:240] Negotiation successful
0927 02:34:00.650445 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":130,"threads_started":1}
I20260927 02:34:00.652199 6290 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42979
I20260927 02:34:00.652398 6290 heartbeater.cc:464] Registering TS with master...
I20260927 02:34:00.652815 6290 heartbeater.cc:511] Master 127.0.98.62:42979 requested a full tablet report, sending...
I20260927 02:34:00.653775 5965 ts_manager.cc:194] Registered new tserver with Master: 5b6d66fca4c34374a324e742db9e1a82 (127.0.98.2:42355)
I20260927 02:34:00.654506 5965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59179
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:34:00.662938 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42979
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:34:00.766438 6295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:34:00.766719 6295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:34:00.766780 6295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:34:00.770814 6295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:34:00.770877 6295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:34:00.770955 6295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:34:00.776098 6295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:34:00.777132 6295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:34:00.778085 6295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:34:00.787063 6301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:00.787113 6295 server_base.cc:1034] running on GCE node
W20260927 02:34:00.787074 6303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:00.787269 6300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:00.787657 6295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:34:00.788218 6295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:34:00.789361 6295 hybrid_clock.cc:648] HybridClock initialized: now 1790476440789346 us; error 28 us; skew 500 ppm
Sep 27 02:34:00 dist-test-slave-grlk krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:34:00.791765 6295 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:34:00.792529 6295 webserver.cc:460] Webserver started at http://127.0.98.3:42453/ using document root <none> and password file <none>
I20260927 02:34:00.793066 6295 fs_manager.cc:359] Metadata directory not provided
I20260927 02:34:00.793135 6295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:34:00.793320 6295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:34:00.795230 6295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/ts-2/data/instance:
uuid: "a393fb3c57b64a418770e0ef2b6fb127"
format_stamp: "Formatted at 2026-09-27 02:34:00 on dist-test-slave-grlk"
server_key: "5ed45c32aac9fbf9f21dafce3e8eaacc"
server_key_iv: "fe54d4629ff129293cd17eb888663764"
server_key_version: "encryptionkey@0"
I20260927 02:34:00.795718 6295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a393fb3c57b64a418770e0ef2b6fb127"
format_stamp: "Formatted at 2026-09-27 02:34:00 on dist-test-slave-grlk"
server_key: "5ed45c32aac9fbf9f21dafce3e8eaacc"
server_key_iv: "fe54d4629ff129293cd17eb888663764"
server_key_version: "encryptionkey@0"
I20260927 02:34:00.799304 6295 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:34:00.801692 6310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:00.802986 6295 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:34:00.803107 6295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/ts-2/wal
uuid: "a393fb3c57b64a418770e0ef2b6fb127"
format_stamp: "Formatted at 2026-09-27 02:34:00 on dist-test-slave-grlk"
server_key: "5ed45c32aac9fbf9f21dafce3e8eaacc"
server_key_iv: "fe54d4629ff129293cd17eb888663764"
server_key_version: "encryptionkey@0"
I20260927 02:34:00.803231 6295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:34:00.829865 6295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:34:00.832617 6295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:34:00.832787 6295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:34:00.833393 6295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:34:00.834194 6295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:34:00.834239 6295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:00.834291 6295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:34:00.834316 6295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:00.842154 6295 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45905
I20260927 02:34:00.842187 6423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45905 every 8 connection(s)
I20260927 02:34:00.842952 6295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:34:00.848094 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 6295
I20260927 02:34:00.848206 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:34:00.848455 392 external_mini_cluster.cc:1428] Setting key 74fe761880e3d1d3d83785e414a480e6
Sep 27 02:34:00 dist-test-slave-grlk krb5kdc[5920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:34:00.855165 6004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:00.844555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41999 (local address 127.0.98.62:42979)
0927 02:34:00.844684 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:00.844687 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:00.845455 (+ 768us) server_negotiation.cc:407] Connection header received
0927 02:34:00.846725 (+ 1270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:00.846728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:00.846780 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:00.846851 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:00.847929 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:00.848514 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:00.849973 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:00.850197 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:00.850281 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:00.852465 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:00.852468 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:00.852471 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:00.852493 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:00.853604 (+ 1111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:00.854030 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:00.854032 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:00.854033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:00.854097 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:00.854349 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:00.854351 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:00.854352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:00.854503 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:00.854571 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:00.854953 (+ 382us) server_negotiation.cc:299] Negotiation successful
0927 02:34:00.855042 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 02:34:00.855904 6426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:00.844896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42979 (local address 127.0.98.3:41999)
0927 02:34:00.845324 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:00.845363 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:00.846530 (+ 1167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:00.847015 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:00.847022 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:00.847350 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:00.847802 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:00.847816 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:00.848621 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:00.848624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:00.849880 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:00.849890 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:00.850321 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:00.850325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:00.850492 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:00.850958 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:00.850973 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:00.852307 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:34:00.853717 (+ 1410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:00.853721 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:00.853732 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:00.853955 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:00.854175 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:00.854177 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:00.854178 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:00.854289 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:00.854568 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:34:00.854573 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:34:00.854782 (+ 209us) client_negotiation.cc:769] Sending connection context
0927 02:34:00.854924 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 02:34:00.855148 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":104,"threads_started":1}
I20260927 02:34:00.856848 6424 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42979
I20260927 02:34:00.857067 6424 heartbeater.cc:464] Registering TS with master...
I20260927 02:34:00.857522 6424 heartbeater.cc:511] Master 127.0.98.62:42979 requested a full tablet report, sending...
I20260927 02:34:00.858417 5965 ts_manager.cc:194] Registered new tserver with Master: a393fb3c57b64a418770e0ef2b6fb127 (127.0.98.3:45905)
I20260927 02:34:00.858952 5965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41999
I20260927 02:34:00.861805 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/krb5kdc/test-admin.keytab.
Sep 27 02:34:00 dist-test-slave-grlk krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:34:00.876590 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 02:34:00.879135 6431 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:00.877934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35042 (local address 127.0.98.1:39079)
0927 02:34:00.878218 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:00.878228 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:00.878253 (+ 25us) server_negotiation.cc:407] Connection header received
0927 02:34:00.878300 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:00.878304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:00.878484 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:00.878595 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:00.878974 (+ 379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35042: BlockingRecv error: recv got EOF from 127.0.0.1:35042 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":78,"threads_started":1}
W20260927 02:34:00.879197 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 02:34:00 dist-test-slave-grlk krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:34:00.880088 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 02:34:00 dist-test-slave-grlk krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:34:00.880889 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 02:34:00 dist-test-slave-grlk krb5kdc[5920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:34:00.889544 6431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:00.881685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35044 (local address 127.0.98.1:39079)
0927 02:34:00.881805 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:00.881808 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:00.881821 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:34:00.881899 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:00.881901 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:00.881937 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:00.882000 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:00.882453 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:00.883209 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:00.883989 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:00.884335 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:00.884406 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:00.886405 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:00.886413 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:00.886425 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:00.886445 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:00.887838 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:00.888218 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:00.888223 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:00.888225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:00.888299 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:00.888607 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:00.888611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:00.888613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:00.888874 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:00.889097 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:00.889230 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 02:34:00.889373 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 02:34:00.892904 6431 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:00.892095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35060 (local address 127.0.98.1:39079)
0927 02:34:00.892203 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:00.892207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:00.892262 (+ 55us) server_negotiation.cc:407] Connection header received
0927 02:34:00.892375 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:00.892383 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:00.892417 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:00.892474 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:00.892826 (+ 352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35060: BlockingRecv error: recv got EOF from 127.0.0.1:35060 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260927 02:34:00.893064 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 02:34:00 dist-test-slave-grlk krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:34:00.893939 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 02:34:00 dist-test-slave-grlk krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:34:00.894711 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 02:34:00 dist-test-slave-grlk krb5kdc[5920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:34:00.901959 6431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:00.895389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35074 (local address 127.0.98.1:39079)
0927 02:34:00.895496 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:00.895500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:00.895566 (+ 66us) server_negotiation.cc:407] Connection header received
0927 02:34:00.895675 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:00.895678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:00.895712 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:00.895766 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:00.896220 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:00.896716 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:00.897485 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:00.897711 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:00.897792 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:00.899606 (+ 1814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:00.899609 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:00.899611 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:00.899629 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:00.900622 (+ 993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:00.900988 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:00.900992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:00.900994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:00.901062 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:00.901283 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:00.901287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:00.901289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:00.901428 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:00.901539 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:00.901731 (+ 192us) server_negotiation.cc:299] Negotiation successful
0927 02:34:00.901823 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260927 02:34:00.904130 6431 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:00.903354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35076 (local address 127.0.98.1:39079)
0927 02:34:00.903475 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:00.903479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:00.903582 (+ 103us) server_negotiation.cc:407] Connection header received
0927 02:34:00.903661 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:00.903663 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:00.903692 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:00.903749 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:00.904045 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35076: BlockingRecv error: recv got EOF from 127.0.0.1:35076 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260927 02:34:00.904239 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476410933508-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 02:34:00 dist-test-slave-grlk krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:34:00.905164 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 02:34:00 dist-test-slave-grlk krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:34:00.905917 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 02:34:00 dist-test-slave-grlk krb5kdc[5920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:34:00.913412 6431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:00.906608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35086 (local address 127.0.98.1:39079)
0927 02:34:00.906698 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:00.906703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:00.906833 (+ 130us) server_negotiation.cc:407] Connection header received
0927 02:34:00.906931 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:00.906933 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:00.906964 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:00.907023 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:00.907443 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:00.907964 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:00.908745 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:00.908982 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:00.909067 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:00.911076 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:00.911079 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:00.911081 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:00.911100 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:00.912121 (+ 1021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:00.912467 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:00.912469 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:00.912471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:00.912530 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:00.912782 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:00.912784 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:00.912785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:00.912923 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:00.913044 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:00.913210 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 02:34:00.913290 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 02:34:00.914319 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 6027
I20260927 02:34:00.919557 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 6161
I20260927 02:34:00.923866 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 6295
I20260927 02:34:00.927986 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 5933
2026-09-27T02:34:00Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1989 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:34:00 dist-test-slave-grlk krb5kdc[6444](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:34:00 dist-test-slave-grlk krb5kdc[6444](info): set up 2 sockets
Sep 27 02:34:00 dist-test-slave-grlk krb5kdc[6444](info): commencing operation
krb5kdc: starting...
W20260927 02:34:01.876878 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/test-user.keytab.
Sep 27 02:34:01 dist-test-slave-grlk krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:34:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:34:01Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 02:34:01.984262 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46055
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46055
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:34:02.093677 6457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:34:02.093931 6457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:34:02.093982 6457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:34:02.097800 6457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:34:02.097846 6457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:34:02.097872 6457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:34:02.097892 6457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:34:02.097918 6457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:34:02.102854 6457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:34:02.103760 6457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:34:02.104686 6457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:34:02.112324 6462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:02.112360 6463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:02.112376 6465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:02.112475 6457 server_base.cc:1034] running on GCE node
I20260927 02:34:02.113131 6457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:34:02.114087 6457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:34:02.115231 6457 hybrid_clock.cc:648] HybridClock initialized: now 1790476442115209 us; error 33 us; skew 500 ppm
Sep 27 02:34:02 dist-test-slave-grlk krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476442, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:34:02.117683 6457 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 02:34:02.118431 6457 webserver.cc:460] Webserver started at http://127.0.98.62:41445/ using document root <none> and password file <none>
I20260927 02:34:02.118912 6457 fs_manager.cc:359] Metadata directory not provided
I20260927 02:34:02.118968 6457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:34:02.119151 6457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:34:02.121027 6457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "aff7cb041f714927ae5a16a8dc279205"
format_stamp: "Formatted at 2026-09-27 02:34:02 on dist-test-slave-grlk"
server_key: "0e1b35a32abbd8e99023ff02f52573c3"
server_key_iv: "a08dd6755edd1496f33fb7629d3e78f6"
server_key_version: "encryptionkey@0"
I20260927 02:34:02.121526 6457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "aff7cb041f714927ae5a16a8dc279205"
format_stamp: "Formatted at 2026-09-27 02:34:02 on dist-test-slave-grlk"
server_key: "0e1b35a32abbd8e99023ff02f52573c3"
server_key_iv: "a08dd6755edd1496f33fb7629d3e78f6"
server_key_version: "encryptionkey@0"
I20260927 02:34:02.124727 6457 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:34:02.126849 6472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:02.127946 6457 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 02:34:02.128048 6457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "aff7cb041f714927ae5a16a8dc279205"
format_stamp: "Formatted at 2026-09-27 02:34:02 on dist-test-slave-grlk"
server_key: "0e1b35a32abbd8e99023ff02f52573c3"
server_key_iv: "a08dd6755edd1496f33fb7629d3e78f6"
server_key_version: "encryptionkey@0"
I20260927 02:34:02.128135 6457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:34:02.144062 6457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:34:02.146713 6457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:34:02.146867 6457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:34:02.152861 6457 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46055
I20260927 02:34:02.153113 6524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46055 every 8 connection(s)
I20260927 02:34:02.153749 6457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:34:02.156236 6525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:34:02.159344 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 6457
I20260927 02:34:02.159468 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:34:02.159754 392 external_mini_cluster.cc:1428] Setting key 24311f890091f2c3ba09d528df0f59e9
I20260927 02:34:02.161630 6525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205: Bootstrap starting.
I20260927 02:34:02.163843 6525 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205: Neither blocks nor log segments found. Creating new log.
I20260927 02:34:02.164510 6525 log.cc:824] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205: Log is configured to *not* fsync() on all Append() calls
I20260927 02:34:02.166378 6525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205: No bootstrap required, opened a new log
Sep 27 02:34:02 dist-test-slave-grlk krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:34:02.169970 6525 raft_consensus.cc:348] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } }
I20260927 02:34:02.170228 6525 raft_consensus.cc:374] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:34:02.170282 6525 raft_consensus.cc:729] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aff7cb041f714927ae5a16a8dc279205, State: Initialized, Role: FOLLOWER
I20260927 02:34:02.170701 6525 consensus_queue.cc:260] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } }
I20260927 02:34:02.170849 6525 raft_consensus.cc:388] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:34:02.170912 6525 raft_consensus.cc:482] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:34:02.170997 6525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:34:02.171881 6525 raft_consensus.cc:504] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } }
I20260927 02:34:02.172173 6525 leader_election.cc:302] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aff7cb041f714927ae5a16a8dc279205; no voters:
I20260927 02:34:02.172420 6525 leader_election.cc:288] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:34:02.172490 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:02.160990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57680 (local address 127.0.98.62:46055)
0927 02:34:02.161364 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:02.161378 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:02.161410 (+ 32us) server_negotiation.cc:407] Connection header received
0927 02:34:02.162009 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:02.162026 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:02.162325 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:02.162586 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:02.163387 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:02.164355 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:02.165342 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:02.165783 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:02.165925 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:02.168382 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:02.168388 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:02.168399 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:02.168422 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:02.169944 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:02.170384 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:02.170390 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:02.170395 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:02.170510 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:02.170759 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:02.170761 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:02.170762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:02.171141 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:02.171281 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:02.171520 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 02:34:02.171766 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":98,"threads_started":1}
I20260927 02:34:02.172513 6530 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:34:02.172676 6530 raft_consensus.cc:686] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205 [term 1 LEADER]: Becoming Leader. State: Replica: aff7cb041f714927ae5a16a8dc279205, State: Running, Role: LEADER
I20260927 02:34:02.172981 6530 consensus_queue.cc:237] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } }
I20260927 02:34:02.173528 6525 sys_catalog.cc:564] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:34:02.174052 6532 sys_catalog.cc:455] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aff7cb041f714927ae5a16a8dc279205. Latest consensus state: current_term: 1 leader_uuid: "aff7cb041f714927ae5a16a8dc279205" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } }
I20260927 02:34:02.174158 6532 sys_catalog.cc:458] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205 [sys.catalog]: This master's current role is: LEADER
I20260927 02:34:02.174712 6531 sys_catalog.cc:455] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aff7cb041f714927ae5a16a8dc279205" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } }
I20260927 02:34:02.174798 6531 sys_catalog.cc:458] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205 [sys.catalog]: This master's current role is: LEADER
I20260927 02:34:02.175577 6536 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:34:02.177904 6536 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:34:02.182413 6536 catalog_manager.cc:1347] Generated new cluster ID: b32f0fff8a2b4f56877e69a9f46fb0b9
I20260927 02:34:02.182468 6536 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:34:02.192902 6536 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:34:02.193714 6536 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:34:02.198662 6536 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205: Generated new TSK 0
I20260927 02:34:02.199183 6536 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 02:34:02.227980 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46055
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:34:02.330178 6551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:34:02.330417 6551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:34:02.330462 6551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:34:02.334447 6551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:34:02.334493 6551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:34:02.334546 6551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:34:02.339337 6551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:34:02.340173 6551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:34:02.341039 6551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:34:02.349876 6557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:02.349905 6556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:02.349844 6559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:02.350013 6551 server_base.cc:1034] running on GCE node
I20260927 02:34:02.350466 6551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:34:02.351063 6551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:34:02.352229 6551 hybrid_clock.cc:648] HybridClock initialized: now 1790476442352211 us; error 28 us; skew 500 ppm
Sep 27 02:34:02 dist-test-slave-grlk krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476442, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:34:02.354729 6551 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 02:34:02.355474 6551 webserver.cc:460] Webserver started at http://127.0.98.1:35515/ using document root <none> and password file <none>
I20260927 02:34:02.355942 6551 fs_manager.cc:359] Metadata directory not provided
I20260927 02:34:02.355996 6551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:34:02.356150 6551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:34:02.358052 6551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-0/data/instance:
uuid: "48c360f547f84f9881723197501c1cc0"
format_stamp: "Formatted at 2026-09-27 02:34:02 on dist-test-slave-grlk"
server_key: "ba61ebfcf93ae2649500ca1c16bcbbba"
server_key_iv: "afc1140d04c933c0bcc111fce19cc0c7"
server_key_version: "encryptionkey@0"
I20260927 02:34:02.358511 6551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-0/wal/instance:
uuid: "48c360f547f84f9881723197501c1cc0"
format_stamp: "Formatted at 2026-09-27 02:34:02 on dist-test-slave-grlk"
server_key: "ba61ebfcf93ae2649500ca1c16bcbbba"
server_key_iv: "afc1140d04c933c0bcc111fce19cc0c7"
server_key_version: "encryptionkey@0"
I20260927 02:34:02.361804 6551 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:34:02.363941 6566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:02.365072 6551 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:34:02.365175 6551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-0/wal
uuid: "48c360f547f84f9881723197501c1cc0"
format_stamp: "Formatted at 2026-09-27 02:34:02 on dist-test-slave-grlk"
server_key: "ba61ebfcf93ae2649500ca1c16bcbbba"
server_key_iv: "afc1140d04c933c0bcc111fce19cc0c7"
server_key_version: "encryptionkey@0"
I20260927 02:34:02.365267 6551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:34:02.378695 6551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:34:02.381135 6551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:34:02.381283 6551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:34:02.381780 6551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:34:02.382515 6551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:34:02.382565 6551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:02.382604 6551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:34:02.382632 6551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:02.390250 6551 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46237
I20260927 02:34:02.390309 6679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46237 every 8 connection(s)
I20260927 02:34:02.391057 6551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:34:02.392774 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 6551
I20260927 02:34:02.392875 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:34:02.393160 392 external_mini_cluster.cc:1428] Setting key 904bc1d6d310c84ebf2ae0363c969190
Sep 27 02:34:02 dist-test-slave-grlk krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476442, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:34:02.403273 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:02.392639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40647 (local address 127.0.98.62:46055)
0927 02:34:02.392775 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:02.392780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:02.393620 (+ 840us) server_negotiation.cc:407] Connection header received
0927 02:34:02.394924 (+ 1304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:02.394928 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:02.394976 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:02.395058 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:02.396082 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:02.396604 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:02.397712 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:02.397945 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:02.398038 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:02.400161 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:02.400165 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:02.400168 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:02.400193 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:02.401586 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:02.401979 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:02.401981 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:02.401982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:02.402047 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:02.402338 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:02.402340 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:02.402341 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:02.402495 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:02.402587 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:02.403060 (+ 473us) server_negotiation.cc:299] Negotiation successful
0927 02:34:02.403154 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260927 02:34:02.404075 6682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:02.393050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.98.1:40647)
0927 02:34:02.393478 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:02.393534 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:02.394736 (+ 1202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:02.395190 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:02.395196 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:02.395511 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:02.395947 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:02.395959 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:02.396717 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:02.396720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:02.397596 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:02.397605 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:02.398066 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:02.398070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:02.398221 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:02.398726 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:02.398740 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:02.400015 (+ 1275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:34:02.401677 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:02.401682 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:02.401691 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:02.401908 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:02.402127 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:02.402130 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:02.402132 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:02.402266 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:02.402587 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:34:02.402592 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:34:02.402845 (+ 253us) client_negotiation.cc:769] Sending connection context
0927 02:34:02.403020 (+ 175us) client_negotiation.cc:240] Negotiation successful
0927 02:34:02.403304 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":123,"threads_started":1}
I20260927 02:34:02.405143 6680 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46055
I20260927 02:34:02.405335 6680 heartbeater.cc:464] Registering TS with master...
I20260927 02:34:02.405745 6680 heartbeater.cc:511] Master 127.0.98.62:46055 requested a full tablet report, sending...
I20260927 02:34:02.407001 6489 ts_manager.cc:194] Registered new tserver with Master: 48c360f547f84f9881723197501c1cc0 (127.0.98.1:46237)
I20260927 02:34:02.408036 6489 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40647
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 02:34:02.415364 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46055
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:34:02.527351 6685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:34:02.527619 6685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:34:02.527678 6685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:34:02.531649 6685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:34:02.531711 6685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:34:02.531786 6685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:34:02.536923 6685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:34:02.537993 6685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:34:02.539404 6685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:34:02.548949 6691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:02.549058 6693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:02.548974 6690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:02.549199 6685 server_base.cc:1034] running on GCE node
I20260927 02:34:02.549767 6685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:34:02.550382 6685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:34:02.551538 6685 hybrid_clock.cc:648] HybridClock initialized: now 1790476442551504 us; error 41 us; skew 500 ppm
Sep 27 02:34:02 dist-test-slave-grlk krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476442, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:34:02.554165 6685 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 02:34:02.554986 6685 webserver.cc:460] Webserver started at http://127.0.98.2:35353/ using document root <none> and password file <none>
I20260927 02:34:02.555492 6685 fs_manager.cc:359] Metadata directory not provided
I20260927 02:34:02.555562 6685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:34:02.555771 6685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:34:02.557760 6685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-1/data/instance:
uuid: "733db7c7c5b54c6495ba5a80d1e50104"
format_stamp: "Formatted at 2026-09-27 02:34:02 on dist-test-slave-grlk"
server_key: "d20e81c4e1f17576a0e9b000a96a231f"
server_key_iv: "8199784bc714a7659b338f5bfc195046"
server_key_version: "encryptionkey@0"
I20260927 02:34:02.558260 6685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-1/wal/instance:
uuid: "733db7c7c5b54c6495ba5a80d1e50104"
format_stamp: "Formatted at 2026-09-27 02:34:02 on dist-test-slave-grlk"
server_key: "d20e81c4e1f17576a0e9b000a96a231f"
server_key_iv: "8199784bc714a7659b338f5bfc195046"
server_key_version: "encryptionkey@0"
I20260927 02:34:02.561825 6685 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:34:02.564124 6700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:02.565353 6685 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:34:02.565483 6685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-1/wal
uuid: "733db7c7c5b54c6495ba5a80d1e50104"
format_stamp: "Formatted at 2026-09-27 02:34:02 on dist-test-slave-grlk"
server_key: "d20e81c4e1f17576a0e9b000a96a231f"
server_key_iv: "8199784bc714a7659b338f5bfc195046"
server_key_version: "encryptionkey@0"
I20260927 02:34:02.565580 6685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:34:02.585577 6685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:34:02.588774 6685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:34:02.588984 6685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:34:02.589573 6685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:34:02.590409 6685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:34:02.590478 6685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:02.590538 6685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:34:02.590569 6685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:02.599321 6685 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37325
I20260927 02:34:02.599350 6813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37325 every 8 connection(s)
I20260927 02:34:02.600222 6685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:34:02.600592 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 6685
I20260927 02:34:02.600698 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:34:02.601020 392 external_mini_cluster.cc:1428] Setting key f824abeecbdb5f5c8ac39a2a83400935
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Sep 27 02:34:02 dist-test-slave-grlk krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476442, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:34:02.617173 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:02.602147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43385 (local address 127.0.98.62:46055)
0927 02:34:02.602258 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:02.602261 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:02.603227 (+ 966us) server_negotiation.cc:407] Connection header received
0927 02:34:02.604694 (+ 1467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:02.604696 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:02.604741 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:02.604814 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:02.606244 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:02.606916 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:02.608572 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:02.608792 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:02.608877 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:02.613806 (+ 4929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:02.613809 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:02.613810 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:02.613834 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:02.615246 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:02.615704 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:02.615708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:02.615710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:02.615784 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:02.616075 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:02.616078 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:02.616080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:02.616298 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:02.616384 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:02.616850 (+ 466us) server_negotiation.cc:299] Negotiation successful
0927 02:34:02.616996 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 02:34:02.618008 6816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:02.602609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.98.2:43385)
0927 02:34:02.603062 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:02.603112 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:02.604465 (+ 1353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:02.605044 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:02.605055 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:02.605491 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:02.606109 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:02.606127 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:02.607048 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:02.607052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:02.608437 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:02.608448 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:02.608961 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:02.608965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:02.609154 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:02.609708 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:02.609728 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:02.613598 (+ 3870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:34:02.615351 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:02.615356 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:02.615367 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:02.615602 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:02.615863 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:02.615866 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:02.615868 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:02.615991 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:02.616390 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:34:02.616397 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:34:02.616624 (+ 227us) client_negotiation.cc:769] Sending connection context
0927 02:34:02.616793 (+ 169us) client_negotiation.cc:240] Negotiation successful
0927 02:34:02.617126 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":108,"threads_started":1}
I20260927 02:34:02.619076 6814 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46055
I20260927 02:34:02.619329 6814 heartbeater.cc:464] Registering TS with master...
I20260927 02:34:02.619892 6814 heartbeater.cc:511] Master 127.0.98.62:46055 requested a full tablet report, sending...
I20260927 02:34:02.620862 6489 ts_manager.cc:194] Registered new tserver with Master: 733db7c7c5b54c6495ba5a80d1e50104 (127.0.98.2:37325)
I20260927 02:34:02.621417 6489 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43385
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 02:34:02.627174 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46055
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:34:02.732575 6819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:34:02.732812 6819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:34:02.732859 6819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:34:02.736756 6819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:34:02.736809 6819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:34:02.736872 6819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:34:02.741817 6819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:34:02.742728 6819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:34:02.743645 6819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:34:02.752121 6825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:02.752127 6824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:02.752174 6827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:02.753022 6819 server_base.cc:1034] running on GCE node
I20260927 02:34:02.753377 6819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:34:02.753955 6819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:34:02.755106 6819 hybrid_clock.cc:648] HybridClock initialized: now 1790476442755090 us; error 28 us; skew 500 ppm
Sep 27 02:34:02 dist-test-slave-grlk krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476442, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:34:02.757597 6819 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 02:34:02.758359 6819 webserver.cc:460] Webserver started at http://127.0.98.3:38455/ using document root <none> and password file <none>
I20260927 02:34:02.758841 6819 fs_manager.cc:359] Metadata directory not provided
I20260927 02:34:02.758894 6819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:34:02.759050 6819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:34:02.760897 6819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-2/data/instance:
uuid: "65d9be8a776a46f6a53142ae0accd7ee"
format_stamp: "Formatted at 2026-09-27 02:34:02 on dist-test-slave-grlk"
server_key: "041d5bad0d12127ed86c06ed62af722f"
server_key_iv: "fbe1a0a96e3fab7592824ce6985bc719"
server_key_version: "encryptionkey@0"
I20260927 02:34:02.761410 6819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-2/wal/instance:
uuid: "65d9be8a776a46f6a53142ae0accd7ee"
format_stamp: "Formatted at 2026-09-27 02:34:02 on dist-test-slave-grlk"
server_key: "041d5bad0d12127ed86c06ed62af722f"
server_key_iv: "fbe1a0a96e3fab7592824ce6985bc719"
server_key_version: "encryptionkey@0"
I20260927 02:34:02.764742 6819 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 02:34:02.767103 6834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:02.768262 6819 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:34:02.768359 6819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-2/wal
uuid: "65d9be8a776a46f6a53142ae0accd7ee"
format_stamp: "Formatted at 2026-09-27 02:34:02 on dist-test-slave-grlk"
server_key: "041d5bad0d12127ed86c06ed62af722f"
server_key_iv: "fbe1a0a96e3fab7592824ce6985bc719"
server_key_version: "encryptionkey@0"
I20260927 02:34:02.768456 6819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:34:02.781188 6819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:34:02.784122 6819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:34:02.784297 6819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:34:02.784816 6819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:34:02.785598 6819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:34:02.785646 6819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:02.785696 6819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:34:02.785727 6819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:02.793560 6819 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43921
I20260927 02:34:02.793617 6947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43921 every 8 connection(s)
I20260927 02:34:02.794385 6819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:34:02.802249 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 6819
I20260927 02:34:02.802366 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:34:02.802618 392 external_mini_cluster.cc:1428] Setting key 2e37718727383854f2462cc748855805
Sep 27 02:34:02 dist-test-slave-grlk krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476442, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:34:02.806792 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:02.796013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51327 (local address 127.0.98.62:46055)
0927 02:34:02.796158 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:02.796162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:02.796886 (+ 724us) server_negotiation.cc:407] Connection header received
0927 02:34:02.798240 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:02.798244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:02.798302 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:02.798390 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:02.799640 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:02.800166 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:02.801282 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:02.801501 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:02.801587 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:02.804015 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:02.804019 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:02.804022 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:02.804047 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:02.805253 (+ 1206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:02.805686 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:02.805688 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:02.805689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:02.805760 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:02.806006 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:02.806008 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:02.806009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:02.806158 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:02.806226 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:02.806596 (+ 370us) server_negotiation.cc:299] Negotiation successful
0927 02:34:02.806680 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 02:34:02.807504 6950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:02.796351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.98.3:51327)
0927 02:34:02.796750 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:02.796788 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:02.797978 (+ 1190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:02.798562 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:02.798569 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:02.798909 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:02.799483 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:02.799496 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:02.800282 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:02.800285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:02.801167 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:02.801176 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:02.801628 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:02.801631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:02.801780 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:02.802357 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:02.802377 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:02.803866 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:34:02.805355 (+ 1489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:02.805359 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:02.805371 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:02.805601 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:02.805826 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:02.805828 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:02.805829 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:02.805942 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:02.806226 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:34:02.806230 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:34:02.806429 (+ 199us) client_negotiation.cc:769] Sending connection context
0927 02:34:02.806566 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 02:34:02.806787 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":104,"threads_started":1}
I20260927 02:34:02.808445 6948 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46055
I20260927 02:34:02.808657 6948 heartbeater.cc:464] Registering TS with master...
I20260927 02:34:02.809131 6948 heartbeater.cc:511] Master 127.0.98.62:46055 requested a full tablet report, sending...
I20260927 02:34:02.809998 6489 ts_manager.cc:194] Registered new tserver with Master: 65d9be8a776a46f6a53142ae0accd7ee (127.0.98.3:43921)
I20260927 02:34:02.810508 6489 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51327
I20260927 02:34:02.815898 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260927 02:34:02.839442 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45743
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46055,127.0.98.61:45743
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:34:02.944145 6953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:34:02.944377 6953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:34:02.944412 6953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:34:02.948488 6953 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:34:02.948545 6953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:34:02.948571 6953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:34:02.948591 6953 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:34:02.948609 6953 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:34:02.953761 6953 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46055,127.0.98.61:45743
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
W20260927 02:34:02.953918 6953 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:46055,127.0.98.61:45743'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 02:34:02.954641 6953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:34:02.955493 6953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:34:02.963331 6958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:02.963374 6959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:02.963464 6953 server_base.cc:1034] running on GCE node
W20260927 02:34:02.963408 6961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:02.964049 6953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:34:02.965047 6953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:34:02.966193 6953 hybrid_clock.cc:648] HybridClock initialized: now 1790476442966174 us; error 34 us; skew 500 ppm
Sep 27 02:34:02 dist-test-slave-grlk krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476442, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:34:02.968706 6953 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 02:34:02.969575 6953 webserver.cc:460] Webserver started at http://127.0.98.61:34673/ using document root <none> and password file <none>
I20260927 02:34:02.970118 6953 fs_manager.cc:359] Metadata directory not provided
I20260927 02:34:02.970191 6953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:34:02.970377 6953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:34:02.972294 6953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-1/data/instance:
uuid: "591735b7214747d5801020bb2bf64abd"
format_stamp: "Formatted at 2026-09-27 02:34:02 on dist-test-slave-grlk"
server_key: "e3d7bad0af2f1d307ec50144606b7efc"
server_key_iv: "b77762c6ecf9d3b909cb18d1ad347071"
server_key_version: "encryptionkey@0"
I20260927 02:34:02.972810 6953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-1/wal/instance:
uuid: "591735b7214747d5801020bb2bf64abd"
format_stamp: "Formatted at 2026-09-27 02:34:02 on dist-test-slave-grlk"
server_key: "e3d7bad0af2f1d307ec50144606b7efc"
server_key_iv: "b77762c6ecf9d3b909cb18d1ad347071"
server_key_version: "encryptionkey@0"
I20260927 02:34:02.976482 6953 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260927 02:34:02.978686 6968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:02.979789 6953 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:34:02.979897 6953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-1/wal
uuid: "591735b7214747d5801020bb2bf64abd"
format_stamp: "Formatted at 2026-09-27 02:34:02 on dist-test-slave-grlk"
server_key: "e3d7bad0af2f1d307ec50144606b7efc"
server_key_iv: "b77762c6ecf9d3b909cb18d1ad347071"
server_key_version: "encryptionkey@0"
I20260927 02:34:02.979979 6953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:34:02.994572 6953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:34:02.997023 6953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:34:02.997184 6953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:34:03.003196 6953 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:45743
I20260927 02:34:03.003422 7020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:45743 every 8 connection(s)
I20260927 02:34:03.004069 6953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-1/data/info.pb
I20260927 02:34:03.004383 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 6953
I20260927 02:34:03.006795 7021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:34:03.010676 7021 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:34:03 dist-test-slave-grlk krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 27 02:34:03 dist-test-slave-grlk krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476442, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:34:03.017676 7024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.005813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60778 (local address 127.0.98.61:45743)
0927 02:34:03.006191 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:03.006205 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:03.006238 (+ 33us) server_negotiation.cc:407] Connection header received
0927 02:34:03.006844 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:03.006865 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:03.007195 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:03.007453 (+ 258us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:03.008273 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.009294 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.010389 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.010811 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.010920 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.013604 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:03.013609 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:03.013617 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:03.013640 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:03.015099 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.015470 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.015474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.015478 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.015604 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.015889 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.015892 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.015893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.016231 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:03.016424 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:03.016606 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 02:34:03.016861 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":117,"threads_started":1}
I20260927 02:34:03.020303 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.012440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57696 (local address 127.0.98.62:46055)
0927 02:34:03.012558 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:03.012563 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:03.012807 (+ 244us) server_negotiation.cc:407] Connection header received
0927 02:34:03.012954 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:03.012958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:03.013005 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:03.013070 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:03.013652 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.014159 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.014973 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.015189 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.015270 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.017319 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:03.017323 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:03.017326 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:03.017352 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:03.018897 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.019269 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.019272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.019274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.019363 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.019602 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.019605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.019606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.019800 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:03.019872 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:03.020103 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 02:34:03.020196 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 02:34:03.020318 7025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.012367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.0.1:57696)
0927 02:34:03.012702 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:03.012738 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:03.012841 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:03.013117 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:03.013122 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:03.013428 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:03.013540 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.013545 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.014259 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.014262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.014870 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.014878 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.015279 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.015282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.015385 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.015828 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:03.015841 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:03.017196 (+ 1355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:34:03.018984 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:03.018988 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:03.018992 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:03.019187 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:03.019439 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:03.019442 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:03.019444 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:03.019522 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:03.019875 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:34:03.019879 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:34:03.019964 (+ 85us) client_negotiation.cc:769] Sending connection context
0927 02:34:03.020064 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 02:34:03.020195 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":103,"threads_started":1}
I20260927 02:34:03.021310 7021 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45743 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:34:03 dist-test-slave-grlk krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476442, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 02:34:03.027562 7025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.021672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45743 (local address 127.0.0.1:60780)
0927 02:34:03.021745 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:03.021757 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:03.021820 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:03.022006 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:03.022008 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:03.022117 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:03.022204 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.022208 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.023219 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.023221 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.023692 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.023697 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.024026 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.024029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.024106 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.024405 (+ 299us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:03.024415 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:03.025437 (+ 1022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:34:03.026583 (+ 1146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:03.026585 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:03.026586 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:03.026752 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:03.026928 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:03.026930 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:03.026931 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:03.026989 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:03.027235 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:34:03.027237 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:34:03.027297 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 02:34:03.027363 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 02:34:03.027448 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21,"mutex_wait_us":2}
I20260927 02:34:03.027623 7024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.021785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60780 (local address 127.0.98.61:45743)
0927 02:34:03.021869 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:03.021872 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:03.021881 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:34:03.021904 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:03.021906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:03.021933 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:03.021988 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:03.022276 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.023145 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.023763 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.023961 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.024021 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.025547 (+ 1526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:03.025550 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:03.025552 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:03.025570 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:03.026508 (+ 938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.026816 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.026818 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.026819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.026874 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.027043 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.027044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.027045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.027179 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:03.027239 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:03.027402 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 02:34:03.027492 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":33}
I20260927 02:34:03.029726 7021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd: Bootstrap starting.
I20260927 02:34:03.031711 7021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd: Neither blocks nor log segments found. Creating new log.
I20260927 02:34:03.032344 7021 log.cc:824] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd: Log is configured to *not* fsync() on all Append() calls
I20260927 02:34:03.033882 7021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd: No bootstrap required, opened a new log
I20260927 02:34:03.036499 7021 raft_consensus.cc:348] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } peers { permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45743 } }
I20260927 02:34:03.036674 7021 raft_consensus.cc:374] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:34:03.036736 7021 raft_consensus.cc:729] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 591735b7214747d5801020bb2bf64abd, State: Initialized, Role: FOLLOWER
I20260927 02:34:03.037130 7021 consensus_queue.cc:260] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } peers { permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45743 } }
I20260927 02:34:03.037495 7027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } peers { permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45743 } } }
I20260927 02:34:03.037592 7027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:34:03.038093 7021 sys_catalog.cc:564] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [sys.catalog]: configured and running, proceeding with master startup.
W20260927 02:34:03.041935 7038 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:34:03.041998 7038 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:34:03.043687 6953 master_runner.cc:428] Detected that this master 591735b7214747d5801020bb2bf64abd is joining an existing cluster
I20260927 02:34:03.043733 6953 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:45743
I20260927 02:34:03.053005 7024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.045881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60794 (local address 127.0.98.61:45743)
0927 02:34:03.045972 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:03.045976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:03.046170 (+ 194us) server_negotiation.cc:407] Connection header received
0927 02:34:03.046255 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:03.046257 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:03.046290 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:03.046348 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:03.046748 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.047482 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.048659 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.049017 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.049110 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.050108 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:03.050111 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:03.050113 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:03.050130 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:03.051547 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.051932 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.051936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.051938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.052023 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.052255 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.052258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.052260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.052413 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:03.052489 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:03.052701 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 02:34:03.052867 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":115,"server-negotiator.queue_time_us":36}
I20260927 02:34:03.053016 7045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.045815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45743 (local address 127.0.0.1:60794)
0927 02:34:03.046101 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:03.046112 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:03.046194 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:03.046371 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:03.046373 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:03.046575 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:03.046655 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.046660 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.047597 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.047601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.048552 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.048562 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.049125 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.049129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.049223 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.049561 (+ 338us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:03.049575 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:03.049980 (+ 405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:34:03.051627 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:03.051630 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:03.051632 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:03.051844 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:03.052104 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:03.052106 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:03.052107 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:03.052179 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:03.052490 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:34:03.052493 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:34:03.052564 (+ 71us) client_negotiation.cc:769] Sending connection context
0927 02:34:03.052645 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 02:34:03.052805 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":53,"spinlock_wait_cycles":2816,"thread_start_us":69,"threads_started":1}
I20260927 02:34:03.053231 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.045529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57706 (local address 127.0.98.62:46055)
0927 02:34:03.045646 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:03.045650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:03.045876 (+ 226us) server_negotiation.cc:407] Connection header received
0927 02:34:03.046062 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:03.046065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:03.046111 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:03.046199 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:03.046784 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.047536 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.048371 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.048585 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.048674 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.049860 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:03.049864 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:03.049866 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:03.049889 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:03.051267 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.051758 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.051761 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.051763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.051840 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.052313 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.052317 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.052319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.052511 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:03.052575 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:03.052965 (+ 390us) server_negotiation.cc:299] Negotiation successful
0927 02:34:03.053074 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 02:34:03.053858 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.045456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.0.1:57706)
0927 02:34:03.045804 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:03.045817 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:03.045940 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:03.046309 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:03.046313 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:03.046526 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:03.046651 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.046658 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.047641 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.047645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.048269 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.048276 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.048791 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.048794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.048961 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.049414 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:03.049424 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:03.049745 (+ 321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:34:03.051376 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:03.051380 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:03.051382 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:03.051654 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:03.052124 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:03.052126 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:03.052128 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:03.052214 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:03.052640 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:34:03.052643 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:34:03.052746 (+ 103us) client_negotiation.cc:769] Sending connection context
0927 02:34:03.053561 (+ 815us) client_negotiation.cc:240] Negotiation successful
0927 02:34:03.053682 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":83,"thread_start_us":77,"threads_started":1}
Sep 27 02:34:03 dist-test-slave-grlk krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476442, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 02:34:03.065269 7024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.057439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60802 (local address 127.0.98.61:45743)
0927 02:34:03.057531 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:03.057534 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:03.057799 (+ 265us) server_negotiation.cc:407] Connection header received
0927 02:34:03.057948 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:03.057950 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:03.057986 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:03.058044 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:03.058688 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.059116 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.060249 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.060434 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.060494 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.062612 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:03.062615 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:03.062617 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:03.062635 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:03.063823 (+ 1188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.064273 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.064275 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.064276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.064336 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.064551 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.064553 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.064554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.064691 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:03.064753 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:03.065099 (+ 346us) server_negotiation.cc:299] Negotiation successful
0927 02:34:03.065174 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 02:34:03.065403 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.057370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45743 (local address 127.0.0.1:60802)
0927 02:34:03.057675 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:03.057726 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:03.057844 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:03.058082 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:03.058088 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:03.058482 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:03.058594 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.058601 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.059198 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.059201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.060159 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.060169 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.060511 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.060514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.060616 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.061053 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:03.061066 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:03.062495 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:34:03.063910 (+ 1415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:03.063917 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:03.063923 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:03.064193 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:03.064400 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:03.064402 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:03.064405 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:03.064484 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:03.064767 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:34:03.064772 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:34:03.064882 (+ 110us) client_negotiation.cc:769] Sending connection context
0927 02:34:03.065064 (+ 182us) client_negotiation.cc:240] Negotiation successful
0927 02:34:03.065238 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":69,"threads_started":1}
I20260927 02:34:03.066584 6489 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:45743: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "591735b7214747d5801020bb2bf64abd"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 45743
}
attrs {
promote: true
}
}
dest_uuid: "aff7cb041f714927ae5a16a8dc279205"
cas_config_opid_index: -1
I20260927 02:34:03.067119 6489 consensus_queue.cc:237] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } peers { permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45743 } attrs { promote: true } }
I20260927 02:34:03.068579 7048 raft_consensus.cc:2932] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 591735b7214747d5801020bb2bf64abd (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } peers { permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45743 } attrs { promote: true } } }
I20260927 02:34:03.069334 6995 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:34:03.070178 7048 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:45743
I20260927 02:34:03.070502 6953 master.cc:537] Master@127.0.98.61:45743 shutting down...
I20260927 02:34:03.070461 7047 sys_catalog.cc:455] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "aff7cb041f714927ae5a16a8dc279205" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } peers { permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45743 } attrs { promote: true } } }
I20260927 02:34:03.070550 7047 sys_catalog.cc:458] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205 [sys.catalog]: This master's current role is: LEADER
I20260927 02:34:03.070554 6995 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [term 1 FOLLOWER]: Refusing update from remote peer aff7cb041f714927ae5a16a8dc279205: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 02:34:03.071190 7047 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45743 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 02:34:03.072738 6474 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260927 02:34:03.073939 6474 consensus_peers.cc:487] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205 -> Peer 591735b7214747d5801020bb2bf64abd (127.0.98.61:45743): Couldn't send request to peer 591735b7214747d5801020bb2bf64abd. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 02:34:03.075623 6953 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:34:03.075863 6953 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:34:03.075953 6953 tablet_replica.cc:330] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd: stopping tablet replica
I20260927 02:34:03.079957 6953 master.cc:552] Master@127.0.98.61:45743 shutdown complete.
I20260927 02:34:03.080004 6953 master_runner.cc:305] Clearing existing system tablet
I20260927 02:34:03.080226 6953 fs_manager.cc:359] Metadata directory not provided
I20260927 02:34:03.080276 6953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260927 02:34:03.081379 6953 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 02:34:03.082727 7053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260927 02:34:03.083098 6953 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:34:03.083246 6953 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:34:03.083278 6953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-1/wal
uuid: "591735b7214747d5801020bb2bf64abd"
format_stamp: "Formatted at 2026-09-27 02:34:02 on dist-test-slave-grlk"
server_key: "e3d7bad0af2f1d307ec50144606b7efc"
server_key_iv: "b77762c6ecf9d3b909cb18d1ad347071"
server_key_version: "encryptionkey@0"
I20260927 02:34:03.085052 6953 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:34:03.087195 6953 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 02:34:03.087280 6953 log.cc:1196] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 02:34:03.087509 6953 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd: Deleting consensus metadata
I20260927 02:34:03.087666 6953 master_runner.cc:315] Copying system tablet from 127.0.98.62:46055
I20260927 02:34:03.088721 6953 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:46055
I20260927 02:34:03.097036 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.089281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57716 (local address 127.0.98.62:46055)
0927 02:34:03.089393 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:03.089398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:03.089617 (+ 219us) server_negotiation.cc:407] Connection header received
0927 02:34:03.089724 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:03.089727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:03.089774 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:03.089851 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:03.090325 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.091114 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.091971 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.092337 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.092434 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.094068 (+ 1634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:03.094071 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:03.094073 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:03.094094 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:03.095443 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.095877 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.095879 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.095881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.095951 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.096215 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.096218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.096220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.096383 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:03.096456 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:03.096753 (+ 297us) server_negotiation.cc:299] Negotiation successful
0927 02:34:03.096841 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 02:34:03.097069 7060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.089189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.0.1:57716)
0927 02:34:03.089521 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:03.089535 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:03.089644 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:03.089879 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:03.089882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:03.090078 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:03.090215 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.090224 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.091251 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.091254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.091843 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.091850 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.092445 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.092449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.092545 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.093491 (+ 946us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:03.093505 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:03.093948 (+ 443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:34:03.095533 (+ 1585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:03.095536 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:03.095538 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:03.095792 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:03.096026 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:03.096029 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:03.096031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:03.096112 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:03.096459 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:34:03.096462 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:34:03.096553 (+ 91us) client_negotiation.cc:769] Sending connection context
0927 02:34:03.096701 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 02:34:03.096831 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":73,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260927 02:34:03.097875 6509 tablet_copy_service.cc:140] P aff7cb041f714927ae5a16a8dc279205: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 591735b7214747d5801020bb2bf64abd ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:57716)
I20260927 02:34:03.098052 6509 tablet_copy_service.cc:161] P aff7cb041f714927ae5a16a8dc279205: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 591735b7214747d5801020bb2bf64abd at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:57716: session id = 591735b7214747d5801020bb2bf64abd-00000000000000000000000000000000
I20260927 02:34:03.098167 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36279
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46055,127.0.98.61:45743,127.0.98.60:36279
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:34:03.099516 6509 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 02:34:03.101125 6953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 02:34:03.103363 6953 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd: tablet copy: Starting download of 0 data blocks...
I20260927 02:34:03.103547 6953 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd: tablet copy: Starting download of 1 WAL segments...
I20260927 02:34:03.105260 6953 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 02:34:03.106992 6509 tablet_copy_service.cc:342] P aff7cb041f714927ae5a16a8dc279205: Request end of tablet copy session 591735b7214747d5801020bb2bf64abd-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:57716
I20260927 02:34:03.107105 6509 tablet_copy_service.cc:433] P aff7cb041f714927ae5a16a8dc279205: ending tablet copy session 591735b7214747d5801020bb2bf64abd-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 591735b7214747d5801020bb2bf64abd
I20260927 02:34:03.114923 6953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:34:03.116595 7065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:03.116907 7062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:03.117100 6953 server_base.cc:1034] running on GCE node
W20260927 02:34:03.125716 7063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:03.125981 6953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:34:03.126271 6953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:34:03.127411 6953 hybrid_clock.cc:648] HybridClock initialized: now 1790476443127393 us; error 34 us; skew 500 ppm
Sep 27 02:34:03 dist-test-slave-grlk krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476443, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:34:03.128913 6953 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 02:34:03.129354 6953 webserver.cc:460] Webserver started at http://127.0.98.61:44515/ using document root <none> and password file <none>
I20260927 02:34:03.129520 6953 fs_manager.cc:359] Metadata directory not provided
I20260927 02:34:03.129557 6953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:34:03.130784 6953 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:34:03.131768 7072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:03.132041 6953 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:34:03.132112 6953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-1/wal
uuid: "591735b7214747d5801020bb2bf64abd"
format_stamp: "Formatted at 2026-09-27 02:34:02 on dist-test-slave-grlk"
server_key: "e3d7bad0af2f1d307ec50144606b7efc"
server_key_iv: "b77762c6ecf9d3b909cb18d1ad347071"
server_key_version: "encryptionkey@0"
I20260927 02:34:03.132192 6953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:34:03.148972 6953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:34:03.150825 6953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:34:03.155285 6953 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:45743
I20260927 02:34:03.155364 7124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:45743 every 8 connection(s)
I20260927 02:34:03.155774 6953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-1/data/info.pb
I20260927 02:34:03.157383 7125 sys_catalog.cc:263] Verifying existing consensus state
I20260927 02:34:03.158439 7125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd: Bootstrap starting.
I20260927 02:34:03.167788 7125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:34:03.168268 7125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd: Bootstrap complete.
I20260927 02:34:03.168829 7125 raft_consensus.cc:348] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } peers { permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45743 } attrs { promote: true } }
I20260927 02:34:03.169059 7125 raft_consensus.cc:729] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 591735b7214747d5801020bb2bf64abd, State: Initialized, Role: LEARNER
I20260927 02:34:03.169214 7125 consensus_queue.cc:260] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } peers { permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45743 } attrs { promote: true } }
I20260927 02:34:03.169783 7125 sys_catalog.cc:564] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:34:03.169901 7128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } peers { permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45743 } attrs { promote: true } } }
I20260927 02:34:03.170061 7128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [sys.catalog]: This master's current role is: LEARNER
I20260927 02:34:03.172540 7139 catalog_manager.cc:1259] Loaded cluster ID: b32f0fff8a2b4f56877e69a9f46fb0b9
I20260927 02:34:03.172623 7139 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd: loading cluster ID for follower catalog manager: success
I20260927 02:34:03.174826 7139 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd: acquiring CA information for follower catalog manager: success
I20260927 02:34:03.176134 7139 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 02:34:03.217267 7061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:34:03.217480 7061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:34:03.217520 7061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:34:03.221397 7061 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:34:03.221442 7061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:34:03.221462 7061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:34:03.221490 7061 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:34:03.221515 7061 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:34:03.226809 7061 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46055,127.0.98.61:45743,127.0.98.60:36279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:34:03.227699 7061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:34:03.228592 7061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:34:03.236359 7145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:03.236442 7147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:03.236544 7144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:03.236603 7061 server_base.cc:1034] running on GCE node
I20260927 02:34:03.237102 7061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:34:03.238049 7061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:34:03.239192 7061 hybrid_clock.cc:648] HybridClock initialized: now 1790476443239168 us; error 39 us; skew 500 ppm
Sep 27 02:34:03 dist-test-slave-grlk krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476443, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:34:03.241518 7061 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 02:34:03.242264 7061 webserver.cc:460] Webserver started at http://127.0.98.60:46497/ using document root <none> and password file <none>
I20260927 02:34:03.242743 7061 fs_manager.cc:359] Metadata directory not provided
I20260927 02:34:03.242803 7061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:34:03.242978 7061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:34:03.244814 7061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-2/data/instance:
uuid: "11b7f1d9d8944fd19085f2cc3c659137"
format_stamp: "Formatted at 2026-09-27 02:34:03 on dist-test-slave-grlk"
server_key: "44f3fbe42aeaf0bc8207ff3a2ee7ddf0"
server_key_iv: "1a6c1f70523ae9f8af71eb522c64550d"
server_key_version: "encryptionkey@0"
I20260927 02:34:03.245339 7061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-2/wal/instance:
uuid: "11b7f1d9d8944fd19085f2cc3c659137"
format_stamp: "Formatted at 2026-09-27 02:34:03 on dist-test-slave-grlk"
server_key: "44f3fbe42aeaf0bc8207ff3a2ee7ddf0"
server_key_iv: "1a6c1f70523ae9f8af71eb522c64550d"
server_key_version: "encryptionkey@0"
I20260927 02:34:03.248545 7061 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:34:03.250638 7154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:03.251703 7061 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 02:34:03.251801 7061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-2/wal
uuid: "11b7f1d9d8944fd19085f2cc3c659137"
format_stamp: "Formatted at 2026-09-27 02:34:03 on dist-test-slave-grlk"
server_key: "44f3fbe42aeaf0bc8207ff3a2ee7ddf0"
server_key_iv: "1a6c1f70523ae9f8af71eb522c64550d"
server_key_version: "encryptionkey@0"
I20260927 02:34:03.251883 7061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:34:03.261868 7061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:34:03.264183 7061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:34:03.264331 7061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:34:03.271059 7061 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:36279
I20260927 02:34:03.271272 7206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:36279 every 8 connection(s)
I20260927 02:34:03.271958 7061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-2/data/info.pb
I20260927 02:34:03.274595 7207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:34:03.277941 7207 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:34:03.279747 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 7061
Sep 27 02:34:03 dist-test-slave-grlk krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476443, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 02:34:03 dist-test-slave-grlk krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 02:34:03.289986 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.279580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57724 (local address 127.0.98.62:46055)
0927 02:34:03.279733 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:03.279738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:03.280450 (+ 712us) server_negotiation.cc:407] Connection header received
0927 02:34:03.281444 (+ 994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:03.281447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:03.281490 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:03.281549 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:03.282349 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.282861 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.283908 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.284187 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.284260 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.286575 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:03.286578 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:03.286581 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:03.286607 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:03.288069 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.288576 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.288579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.288581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.288662 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.289022 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.289025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.289026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.289227 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:03.289309 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:03.289732 (+ 423us) server_negotiation.cc:299] Negotiation successful
0927 02:34:03.289834 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 02:34:03.290719 7209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.279882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.0.1:57724)
0927 02:34:03.280313 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:03.280360 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:03.281254 (+ 894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:03.281608 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:03.281616 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:03.282010 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:03.282219 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.282235 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.282971 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.282975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.283808 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.283816 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.284299 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.284303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.284465 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.285015 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:03.285030 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:03.286444 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:34:03.288170 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:03.288176 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:03.288182 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:03.288492 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:03.288742 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:03.288744 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:03.288746 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:03.288895 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:03.289317 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:34:03.289321 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:34:03.289519 (+ 198us) client_negotiation.cc:769] Sending connection context
0927 02:34:03.289656 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 02:34:03.289928 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":95,"threads_started":1}
I20260927 02:34:03.290720 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.280422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34346 (local address 127.0.98.60:36279)
0927 02:34:03.280749 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:03.280759 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:03.280792 (+ 33us) server_negotiation.cc:407] Connection header received
0927 02:34:03.281270 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:03.281290 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:03.281500 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:03.281662 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:03.282275 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.283291 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.284100 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.284514 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.284610 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.287015 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:03.287020 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:03.287030 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:03.287054 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:03.288547 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.288976 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.288982 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.288984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.289064 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.289297 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.289300 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.289301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.289521 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:03.289677 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:03.289882 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 02:34:03.290001 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":90,"threads_started":1}
I20260927 02:34:03.291800 7207 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45743 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:34:03 dist-test-slave-grlk krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476443, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 02:34:03.298628 7209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.292156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45743 (local address 127.0.0.1:60814)
0927 02:34:03.292303 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:03.292315 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:03.292377 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:03.292669 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:03.292671 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:03.292777 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:03.292855 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.292859 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.293697 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.293699 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.294192 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.294198 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.294534 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.294537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.294599 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.294913 (+ 314us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:03.294923 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:03.295989 (+ 1066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:34:03.297455 (+ 1466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:03.297458 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:03.297460 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:03.297626 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:03.297848 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:03.297851 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:03.297853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:03.297921 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:03.298228 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:34:03.298231 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:34:03.298309 (+ 78us) client_negotiation.cc:769] Sending connection context
0927 02:34:03.298387 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 02:34:03.298494 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260927 02:34:03.298700 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.292270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60814 (local address 127.0.98.61:45743)
0927 02:34:03.292488 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:03.292497 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:03.292509 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:34:03.292541 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:03.292542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:03.292578 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:03.292657 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:03.292952 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.293604 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.294267 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.294469 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.294532 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.296109 (+ 1577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:03.296112 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:03.296114 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:03.296135 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:03.297362 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.297704 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.297707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.297709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.297774 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.297992 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.297995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.297997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.298149 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:03.298227 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:03.298443 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 02:34:03.298546 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":69,"threads_started":1}
I20260927 02:34:03.299242 7207 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36279 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:34:03 dist-test-slave-grlk krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476443, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 02:34:03.305778 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.299622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34356 (local address 127.0.98.60:36279)
0927 02:34:03.299707 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:03.299710 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:03.299722 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:34:03.299805 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:03.299808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:03.299840 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:03.299898 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:03.300183 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.300646 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.301334 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.301561 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.301638 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.303258 (+ 1620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:03.303261 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:03.303262 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:03.303281 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:03.304280 (+ 999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.304672 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.304674 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.304675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.304733 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.305099 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.305101 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.305102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.305246 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:03.305314 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:03.305525 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 02:34:03.305650 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":34}
I20260927 02:34:03.305778 7209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.299574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36279 (local address 127.0.0.1:34356)
0927 02:34:03.299649 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:03.299671 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:03.299736 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:03.299915 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:03.299917 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:03.300036 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:03.300088 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.300092 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.300733 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.300736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.301250 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.301258 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.301643 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.301646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.301741 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.302092 (+ 351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:03.302102 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:03.303149 (+ 1047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:34:03.304375 (+ 1226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:03.304378 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:03.304381 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:03.304584 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:03.304866 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:03.304869 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:03.304871 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:03.304996 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:03.305321 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:34:03.305324 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:34:03.305403 (+ 79us) client_negotiation.cc:769] Sending connection context
0927 02:34:03.305493 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 02:34:03.305604 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":20}
I20260927 02:34:03.308627 7207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137: Bootstrap starting.
I20260927 02:34:03.310725 7207 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137: Neither blocks nor log segments found. Creating new log.
I20260927 02:34:03.311378 7207 log.cc:824] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137: Log is configured to *not* fsync() on all Append() calls
I20260927 02:34:03.313058 7207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137: No bootstrap required, opened a new log
I20260927 02:34:03.316184 7207 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } peers { permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45743 } } peers { permanent_uuid: "11b7f1d9d8944fd19085f2cc3c659137" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36279 } }
I20260927 02:34:03.316380 7207 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:34:03.316431 7207 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11b7f1d9d8944fd19085f2cc3c659137, State: Initialized, Role: FOLLOWER
I20260927 02:34:03.316825 7207 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } peers { permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45743 } } peers { permanent_uuid: "11b7f1d9d8944fd19085f2cc3c659137" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36279 } }
I20260927 02:34:03.317238 7213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } peers { permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45743 } } peers { permanent_uuid: "11b7f1d9d8944fd19085f2cc3c659137" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36279 } } }
I20260927 02:34:03.317341 7213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:34:03.317865 7207 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 02:34:03.322404 7224 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:34:03.322496 7224 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:34:03.326287 7061 master_runner.cc:428] Detected that this master 11b7f1d9d8944fd19085f2cc3c659137 is joining an existing cluster
I20260927 02:34:03.326362 7061 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:36279
I20260927 02:34:03.338877 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.328885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57740 (local address 127.0.98.62:46055)
0927 02:34:03.329020 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:03.329025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:03.329649 (+ 624us) server_negotiation.cc:407] Connection header received
0927 02:34:03.329854 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:03.329858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:03.329910 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:03.329962 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:03.330611 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.331434 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.332797 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.333136 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.333221 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.335061 (+ 1840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:03.335064 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:03.335066 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:03.335099 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:03.336897 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.337575 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.337579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.337580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.337681 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.338009 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.338011 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.338012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.338206 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:03.338287 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:03.338623 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 02:34:03.338749 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 02:34:03.338872 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.329033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.0.1:57740)
0927 02:34:03.329480 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:03.329497 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:03.329735 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:03.330102 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:03.330105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:03.330360 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:03.330489 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.330495 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.331600 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.331603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.332659 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.332670 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.333399 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.333402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.333486 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.334155 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:03.334172 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:03.334825 (+ 653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:34:03.337108 (+ 2283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:03.337112 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:03.337115 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:03.337435 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:03.337778 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:03.337781 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:03.337783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:03.337875 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:03.338288 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:34:03.338291 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:34:03.338399 (+ 108us) client_negotiation.cc:769] Sending connection context
0927 02:34:03.338503 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 02:34:03.338665 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":66,"thread_start_us":92,"threads_started":1}
I20260927 02:34:03.343889 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 6551
Sep 27 02:34:03 dist-test-slave-grlk krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476442, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 02:34:03.351418 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46237
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:46055
--webserver_port=35515
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:46055,127.0.98.61:45743
--tserver_master_addrs=127.0.98.62:46055,127.0.98.61:45743,127.0.98.60:36279 with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:34:03.361990 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.329801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45743 (local address 127.0.0.1:60816)
0927 02:34:03.330106 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:03.330117 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:03.330256 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:03.330584 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:03.330586 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:03.330770 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:03.330879 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.330886 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.331901 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.331904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.332864 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.332872 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.333657 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.333673 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.333874 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.357529 (+ 23655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:03.357551 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:03.358080 (+ 529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:34:03.360093 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:03.360097 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:03.360099 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:03.360426 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:03.360757 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:03.360760 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:03.360762 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:03.360850 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:03.361380 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:34:03.361383 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:34:03.361513 (+ 130us) client_negotiation.cc:769] Sending connection context
0927 02:34:03.361615 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 02:34:03.361774 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":128,"thread_start_us":81,"threads_started":1}
I20260927 02:34:03.362421 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.343447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36279 (local address 127.0.0.1:34382)
0927 02:34:03.343637 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:03.343652 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:03.343730 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:03.344066 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:03.344070 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:03.344281 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:03.344400 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.344407 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.346006 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.346010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.346955 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.346970 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.347738 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.347742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.347853 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.348524 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:03.348540 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:03.350198 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:34:03.360821 (+ 10623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:03.360825 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:03.360827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:03.361127 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:03.361427 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:03.361430 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:03.361432 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:03.361516 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:03.361932 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:34:03.361935 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:34:03.362033 (+ 98us) client_negotiation.cc:769] Sending connection context
0927 02:34:03.362107 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 02:34:03.362258 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260927 02:34:03.362752 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.329592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60816 (local address 127.0.98.61:45743)
0927 02:34:03.329712 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:03.329715 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:03.330200 (+ 485us) server_negotiation.cc:407] Connection header received
0927 02:34:03.330356 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:03.330359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:03.330409 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:03.330461 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:03.331005 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.331756 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.333130 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.333438 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.333508 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.358331 (+ 24823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:03.358337 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:03.358340 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:03.358375 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:03.359940 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.360560 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.360564 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.360566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.360655 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.360995 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.360998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.361000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.361242 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:03.361334 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:03.362466 (+ 1132us) server_negotiation.cc:299] Negotiation successful
0927 02:34:03.362600 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 02:34:03.362972 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.343461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34382 (local address 127.0.98.60:36279)
0927 02:34:03.343782 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:03.343790 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:03.343803 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:34:03.343870 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:03.343873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:03.343925 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:03.344708 (+ 783us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:03.344900 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.345838 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.347129 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.347517 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.347717 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.350446 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:03.350450 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:03.350454 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:03.350489 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:03.360651 (+ 10162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.361248 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.361251 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.361253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.361331 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.361615 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.361618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.361620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.361827 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:03.362606 (+ 779us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:03.362691 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 02:34:03.362807 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":141,"threads_started":1}
I20260927 02:34:03.364171 6489 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:36279: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "11b7f1d9d8944fd19085f2cc3c659137"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 36279
}
attrs {
promote: true
}
}
dest_uuid: "aff7cb041f714927ae5a16a8dc279205"
cas_config_opid_index: 5
I20260927 02:34:03.365100 6489 consensus_queue.cc:237] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } peers { permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45743 } attrs { promote: true } } peers { permanent_uuid: "11b7f1d9d8944fd19085f2cc3c659137" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36279 } attrs { promote: true } }
I20260927 02:34:03.366331 7048 raft_consensus.cc:2932] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 11b7f1d9d8944fd19085f2cc3c659137 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } peers { permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45743 } attrs { promote: true } } peers { permanent_uuid: "11b7f1d9d8944fd19085f2cc3c659137" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36279 } attrs { promote: true } } }
I20260927 02:34:03.366701 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.328861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34368 (local address 127.0.98.60:36279)
0927 02:34:03.328995 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:03.328999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:03.329390 (+ 391us) server_negotiation.cc:407] Connection header received
0927 02:34:03.329548 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:03.329552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:03.329589 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:03.329634 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:03.330253 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.331058 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.332978 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.333528 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.333759 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.363531 (+ 29772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:03.363534 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:03.363537 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:03.363556 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:03.365169 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.365637 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.365640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.365642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.365705 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.365922 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.365925 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.365926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.366108 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:03.366167 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:03.366436 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 02:34:03.366554 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":44}
I20260927 02:34:03.367188 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.328866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36279 (local address 127.0.0.1:34368)
0927 02:34:03.329309 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:03.329325 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:03.329455 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:03.329778 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:03.329781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:03.330006 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:03.330139 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.330147 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.331209 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.331212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.332811 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.332821 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.333834 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.333837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.333914 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.358703 (+ 24789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:03.358718 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:03.363386 (+ 4668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:34:03.365313 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:03.365316 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:03.365318 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:03.365547 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:03.365784 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:03.365787 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:03.365789 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:03.365849 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:03.366219 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:34:03.366221 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:34:03.366302 (+ 81us) client_negotiation.cc:769] Sending connection context
0927 02:34:03.366898 (+ 596us) client_negotiation.cc:240] Negotiation successful
0927 02:34:03.367028 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":31,"thread_start_us":248,"threads_started":1}
I20260927 02:34:03.367781 7048 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:36279
I20260927 02:34:03.367942 7048 sys_catalog.cc:455] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "aff7cb041f714927ae5a16a8dc279205" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } peers { permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45743 } attrs { promote: true } } peers { permanent_uuid: "11b7f1d9d8944fd19085f2cc3c659137" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36279 } attrs { promote: true } } }
I20260927 02:34:03.368034 7048 sys_catalog.cc:458] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205 [sys.catalog]: This master's current role is: LEADER
I20260927 02:34:03.368438 7061 master.cc:537] Master@127.0.98.60:36279 shutting down...
W20260927 02:34:03.370146 6476 consensus_peers.cc:487] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205 -> Peer 11b7f1d9d8944fd19085f2cc3c659137 (127.0.98.60:36279): Couldn't send request to peer 11b7f1d9d8944fd19085f2cc3c659137. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 02:34:03.370878 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.366264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45743 (local address 127.0.0.1:60824)
0927 02:34:03.366677 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:03.366692 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:03.366775 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:03.367131 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:03.367134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:03.367150 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:34:03.367245 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.367250 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.368211 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.368215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.369421 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.369430 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.370464 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.370468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.370563 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.370613 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 02:34:03.370688 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 02:34:03.370764 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347}
I20260927 02:34:03.370918 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.366473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60824 (local address 127.0.98.61:45743)
0927 02:34:03.366594 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:03.366598 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:03.366849 (+ 251us) server_negotiation.cc:407] Connection header received
0927 02:34:03.366889 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:03.366891 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:03.366945 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:03.367022 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:34:03.367359 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.368061 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.369545 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.370136 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.370242 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.370401 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:03.370711 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 02:34:03.370817 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 02:34:03.371423 7099 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [term 1 LEARNER]: Refusing update from remote peer aff7cb041f714927ae5a16a8dc279205: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 02:34:03.372161 7047 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45743 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 02:34:03.374238 7099 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 11b7f1d9d8944fd19085f2cc3c659137 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } peers { permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45743 } attrs { promote: true } } peers { permanent_uuid: "11b7f1d9d8944fd19085f2cc3c659137" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36279 } attrs { promote: true } } }
I20260927 02:34:03.376331 7128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [sys.catalog]: SysCatalogTable state changed. Reason: New leader aff7cb041f714927ae5a16a8dc279205. Latest consensus state: current_term: 1 leader_uuid: "aff7cb041f714927ae5a16a8dc279205" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } peers { permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45743 } attrs { promote: true } } peers { permanent_uuid: "11b7f1d9d8944fd19085f2cc3c659137" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36279 } attrs { promote: true } } }
I20260927 02:34:03.376438 7128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [sys.catalog]: This master's current role is: LEARNER
I20260927 02:34:03.376518 7128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aff7cb041f714927ae5a16a8dc279205" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } peers { permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45743 } attrs { promote: true } } peers { permanent_uuid: "11b7f1d9d8944fd19085f2cc3c659137" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36279 } attrs { promote: true } } }
I20260927 02:34:03.376576 7128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [sys.catalog]: This master's current role is: LEARNER
I20260927 02:34:03.377242 7047 raft_consensus.cc:1050] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205: attempting to promote NON_VOTER 591735b7214747d5801020bb2bf64abd to VOTER
I20260927 02:34:03.377661 7047 consensus_queue.cc:237] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } peers { permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45743 } attrs { promote: false } } peers { permanent_uuid: "11b7f1d9d8944fd19085f2cc3c659137" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36279 } attrs { promote: true } }
I20260927 02:34:03.378557 7061 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260927 02:34:03.378857 7061 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260927 02:34:03.378954 7061 tablet_replica.cc:330] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137: stopping tablet replica
W20260927 02:34:03.380102 6476 consensus_peers.cc:487] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205 -> Peer 11b7f1d9d8944fd19085f2cc3c659137 (127.0.98.60:36279): Couldn't send request to peer 11b7f1d9d8944fd19085f2cc3c659137. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 02:34:03.380789 7099 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [term 1 LEARNER]: Refusing update from remote peer aff7cb041f714927ae5a16a8dc279205: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 02:34:03.381201 7048 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45743 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 02:34:03.383266 7048 raft_consensus.cc:2932] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 591735b7214747d5801020bb2bf64abd (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } peers { permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45743 } attrs { promote: false } } peers { permanent_uuid: "11b7f1d9d8944fd19085f2cc3c659137" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36279 } attrs { promote: true } } }
I20260927 02:34:03.384615 7048 sys_catalog.cc:455] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "aff7cb041f714927ae5a16a8dc279205" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } peers { permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45743 } attrs { promote: false } } peers { permanent_uuid: "11b7f1d9d8944fd19085f2cc3c659137" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36279 } attrs { promote: true } } }
I20260927 02:34:03.384727 7048 sys_catalog.cc:458] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205 [sys.catalog]: This master's current role is: LEADER
I20260927 02:34:03.384837 7048 sys_catalog.cc:455] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aff7cb041f714927ae5a16a8dc279205" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } peers { permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45743 } attrs { promote: false } } peers { permanent_uuid: "11b7f1d9d8944fd19085f2cc3c659137" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36279 } attrs { promote: true } } }
I20260927 02:34:03.384907 7048 sys_catalog.cc:458] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205 [sys.catalog]: This master's current role is: LEADER
I20260927 02:34:03.385399 7098 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 591735b7214747d5801020bb2bf64abd (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } peers { permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45743 } attrs { promote: false } } peers { permanent_uuid: "11b7f1d9d8944fd19085f2cc3c659137" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36279 } attrs { promote: true } } }
I20260927 02:34:03.387017 7236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aff7cb041f714927ae5a16a8dc279205" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } peers { permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45743 } attrs { promote: false } } peers { permanent_uuid: "11b7f1d9d8944fd19085f2cc3c659137" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36279 } attrs { promote: true } } }
I20260927 02:34:03.387132 7236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:34:03.390410 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.390038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36279 (local address 127.0.0.1:34392)
0927 02:34:03.390254 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:03.390323 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 02:34:03.395310 7061 master.cc:552] Master@127.0.98.60:36279 shutdown complete.
I20260927 02:34:03.395370 7061 master_runner.cc:305] Clearing existing system tablet
I20260927 02:34:03.395618 7061 fs_manager.cc:359] Metadata directory not provided
I20260927 02:34:03.395673 7061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:34:03.396924 7061 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:34:03.397802 7239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:03.398031 7061 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:34:03.398154 7061 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:34:03.398185 7061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-2/wal
uuid: "11b7f1d9d8944fd19085f2cc3c659137"
format_stamp: "Formatted at 2026-09-27 02:34:03 on dist-test-slave-grlk"
server_key: "44f3fbe42aeaf0bc8207ff3a2ee7ddf0"
server_key_iv: "1a6c1f70523ae9f8af71eb522c64550d"
server_key_version: "encryptionkey@0"
I20260927 02:34:03.400170 7061 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:34:03.402422 7061 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 02:34:03.402513 7061 log.cc:1196] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 02:34:03.402729 7061 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137: Deleting consensus metadata
I20260927 02:34:03.402870 7061 master_runner.cc:315] Copying system tablet from 127.0.98.62:46055
I20260927 02:34:03.403921 7061 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:46055
I20260927 02:34:03.412488 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.404525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57752 (local address 127.0.98.62:46055)
0927 02:34:03.404644 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:03.404648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:03.404821 (+ 173us) server_negotiation.cc:407] Connection header received
0927 02:34:03.404961 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:03.404964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:03.405017 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:03.405108 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:03.405590 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.406430 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.407577 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.407909 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.408010 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.409262 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:03.409266 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:03.409269 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:03.409299 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:03.410906 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.411346 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.411350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.411352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.411434 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.411689 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.411692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.411693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.411888 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:03.411977 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:03.412225 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 02:34:03.412329 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 02:34:03.412495 7245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.404431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.0.1:57752)
0927 02:34:03.404734 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:03.404747 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:03.404854 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:03.405136 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:03.405139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:03.405342 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:03.405485 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.405492 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.406553 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.406556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.407453 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.407464 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.408017 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.408021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.408114 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.408649 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:03.408663 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:03.409104 (+ 441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:34:03.411009 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:03.411012 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:03.411015 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:03.411254 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:03.411522 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:03.411524 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:03.411526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:03.411608 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:03.411976 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:34:03.411979 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:34:03.412072 (+ 93us) client_negotiation.cc:769] Sending connection context
0927 02:34:03.412172 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 02:34:03.412305 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":95,"threads_started":1}
I20260927 02:34:03.413303 6509 tablet_copy_service.cc:140] P aff7cb041f714927ae5a16a8dc279205: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 11b7f1d9d8944fd19085f2cc3c659137 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:57752)
I20260927 02:34:03.413399 6509 tablet_copy_service.cc:161] P aff7cb041f714927ae5a16a8dc279205: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 11b7f1d9d8944fd19085f2cc3c659137 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:57752: session id = 11b7f1d9d8944fd19085f2cc3c659137-00000000000000000000000000000000
I20260927 02:34:03.414413 6509 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 02:34:03.415522 7061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 02:34:03.417695 7061 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137: tablet copy: Starting download of 0 data blocks...
I20260927 02:34:03.417850 7061 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137: tablet copy: Starting download of 1 WAL segments...
I20260927 02:34:03.419189 7061 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 02:34:03.420710 6509 tablet_copy_service.cc:342] P aff7cb041f714927ae5a16a8dc279205: Request end of tablet copy session 11b7f1d9d8944fd19085f2cc3c659137-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:57752
I20260927 02:34:03.420785 6509 tablet_copy_service.cc:433] P aff7cb041f714927ae5a16a8dc279205: ending tablet copy session 11b7f1d9d8944fd19085f2cc3c659137-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 11b7f1d9d8944fd19085f2cc3c659137
I20260927 02:34:03.430956 7061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:34:03.432632 7249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:03.433071 7246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:03.433176 7247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:03.434072 7061 server_base.cc:1034] running on GCE node
I20260927 02:34:03.434244 7061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:34:03.434525 7061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:34:03.435665 7061 hybrid_clock.cc:648] HybridClock initialized: now 1790476443435654 us; error 27 us; skew 500 ppm
Sep 27 02:34:03 dist-test-slave-grlk krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476443, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:34:03.437263 7061 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 02:34:03.437664 7061 webserver.cc:460] Webserver started at http://127.0.98.60:33715/ using document root <none> and password file <none>
I20260927 02:34:03.437847 7061 fs_manager.cc:359] Metadata directory not provided
I20260927 02:34:03.437886 7061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:34:03.439087 7061 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:34:03.440016 7256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:03.440259 7061 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:34:03.440333 7061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-2/wal
uuid: "11b7f1d9d8944fd19085f2cc3c659137"
format_stamp: "Formatted at 2026-09-27 02:34:03 on dist-test-slave-grlk"
server_key: "44f3fbe42aeaf0bc8207ff3a2ee7ddf0"
server_key_iv: "1a6c1f70523ae9f8af71eb522c64550d"
server_key_version: "encryptionkey@0"
I20260927 02:34:03.440410 7061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:34:03.455909 7061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:34:03.458009 7061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:34:03.462908 7061 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:36279
I20260927 02:34:03.463002 7308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:36279 every 8 connection(s)
I20260927 02:34:03.463465 7061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/master-2/data/info.pb
I20260927 02:34:03.465137 7309 sys_catalog.cc:263] Verifying existing consensus state
I20260927 02:34:03.466164 7309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137: Bootstrap starting.
I20260927 02:34:03.477145 7309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:34:03.477393 7309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137: Bootstrap complete.
I20260927 02:34:03.477887 7309 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } peers { permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45743 } attrs { promote: false } } peers { permanent_uuid: "11b7f1d9d8944fd19085f2cc3c659137" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36279 } attrs { promote: true } }
I20260927 02:34:03.478020 7309 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 11b7f1d9d8944fd19085f2cc3c659137, State: Initialized, Role: LEARNER
I20260927 02:34:03.478127 7309 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } peers { permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45743 } attrs { promote: false } } peers { permanent_uuid: "11b7f1d9d8944fd19085f2cc3c659137" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36279 } attrs { promote: true } }
I20260927 02:34:03.478401 7312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } peers { permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45743 } attrs { promote: false } } peers { permanent_uuid: "11b7f1d9d8944fd19085f2cc3c659137" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36279 } attrs { promote: true } } }
I20260927 02:34:03.478540 7309 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:34:03.478575 7312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137 [sys.catalog]: This master's current role is: LEARNER
I20260927 02:34:03.481628 7323 catalog_manager.cc:1259] Loaded cluster ID: b32f0fff8a2b4f56877e69a9f46fb0b9
I20260927 02:34:03.481696 7323 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137: loading cluster ID for follower catalog manager: success
I20260927 02:34:03.483767 7323 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137: acquiring CA information for follower catalog manager: success
I20260927 02:34:03.485069 7323 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 02:34:03.488878 7234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:34:03.489131 7234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:34:03.489198 7234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:34:03.493086 7234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:34:03.493136 7234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:34:03.493218 7234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:34:03.498272 7234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46237
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35515
--tserver_master_addrs=127.0.98.62:46055,127.0.98.61:45743,127.0.98.60:36279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:34:03.499251 7234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:34:03.500190 7234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:34:03.510079 7234 server_base.cc:1034] running on GCE node
W20260927 02:34:03.510005 7329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:03.510038 7328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:03.510005 7331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:03.510666 7234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:34:03.511240 7234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:34:03.512387 7234 hybrid_clock.cc:648] HybridClock initialized: now 1790476443512369 us; error 28 us; skew 500 ppm
Sep 27 02:34:03 dist-test-slave-grlk krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476443, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:34:03.514889 7234 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 02:34:03.515692 7234 webserver.cc:460] Webserver started at http://127.0.98.1:35515/ using document root <none> and password file <none>
I20260927 02:34:03.516193 7234 fs_manager.cc:359] Metadata directory not provided
I20260927 02:34:03.516264 7234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:34:03.520521 7234 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 02:34:03.522783 7338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:03.524019 7234 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:34:03.524129 7234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-0/wal
uuid: "48c360f547f84f9881723197501c1cc0"
format_stamp: "Formatted at 2026-09-27 02:34:02 on dist-test-slave-grlk"
server_key: "ba61ebfcf93ae2649500ca1c16bcbbba"
server_key_iv: "afc1140d04c933c0bcc111fce19cc0c7"
server_key_version: "encryptionkey@0"
I20260927 02:34:03.524709 7234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:34:03.539001 7234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:34:03.542181 7234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:34:03.542348 7234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:34:03.542904 7234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:34:03.543730 7234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:34:03.543774 7234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:03.543816 7234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:34:03.543844 7234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:03.552605 7234 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46237
I20260927 02:34:03.552634 7451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46237 every 8 connection(s)
I20260927 02:34:03.553578 7234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:34:03.559315 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 7234
I20260927 02:34:03.559499 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 6685
Sep 27 02:34:03 dist-test-slave-grlk krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476443, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 02:34:03 dist-test-slave-grlk krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476443, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 02:34:03 dist-test-slave-grlk krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476443, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 02:34:03.567942 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37325
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:46055
--webserver_port=35353
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:46055,127.0.98.61:45743
--tserver_master_addrs=127.0.98.62:46055,127.0.98.61:45743,127.0.98.60:36279 with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:34:03.568661 7456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.555671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45239 (local address 127.0.98.60:36279)
0927 02:34:03.555921 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:03.555927 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:03.556494 (+ 567us) server_negotiation.cc:407] Connection header received
0927 02:34:03.557317 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:03.557320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:03.557366 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:03.557435 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:03.558660 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.559661 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.560831 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.561212 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.561303 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.564783 (+ 3480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:03.564787 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:03.564790 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:03.564818 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:03.566420 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.566982 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.566985 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.566987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.567067 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.567433 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.567436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.567438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.567647 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:03.567711 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:03.568362 (+ 651us) server_negotiation.cc:299] Negotiation successful
0927 02:34:03.568491 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":84,"threads_started":1}
I20260927 02:34:03.568758 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.556886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48291 (local address 127.0.98.61:45743)
0927 02:34:03.557016 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:03.557020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:03.557035 (+ 15us) server_negotiation.cc:407] Connection header received
0927 02:34:03.557210 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:03.557214 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:03.557267 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:03.557337 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:03.558636 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.559444 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.561710 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.562397 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.562471 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.565512 (+ 3041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:03.565516 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:03.565518 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:03.565546 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:03.566973 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.567378 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.567382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.567383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.567458 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.567744 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.567747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.567748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.567939 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:03.568009 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:03.568523 (+ 514us) server_negotiation.cc:299] Negotiation successful
0927 02:34:03.568620 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 02:34:03.569177 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.555208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48967 (local address 127.0.98.62:46055)
0927 02:34:03.555340 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:03.555344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:03.556243 (+ 899us) server_negotiation.cc:407] Connection header received
0927 02:34:03.557260 (+ 1017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:03.557263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:03.557316 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:03.557367 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:03.558655 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.559242 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.561005 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.561329 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.561396 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.565309 (+ 3913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:03.565313 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:03.565316 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:03.565342 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:03.567314 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.567735 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.567739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.567741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.567818 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.568229 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.568232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.568234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.568429 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:03.568499 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:03.568890 (+ 391us) server_negotiation.cc:299] Negotiation successful
0927 02:34:03.569020 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 02:34:03.570279 7457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.555581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.98.1:48967)
0927 02:34:03.556079 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:03.556128 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:03.557015 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:03.557505 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:03.557508 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:03.557929 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:03.558524 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.558530 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.559404 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.559407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.560856 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.560867 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.561539 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.561542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.561727 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.562359 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:03.562378 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:03.565170 (+ 2792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:34:03.567420 (+ 2250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:03.567423 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:03.567425 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:03.567630 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:03.567912 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:03.567915 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:03.567917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:03.567995 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:03.568556 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:34:03.568559 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:34:03.568635 (+ 76us) client_negotiation.cc:769] Sending connection context
0927 02:34:03.569397 (+ 762us) client_negotiation.cc:240] Negotiation successful
0927 02:34:03.569541 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":72,"spinlock_wait_cycles":14976,"thread_start_us":596,"threads_started":1}
I20260927 02:34:03.570279 7458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.555581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36279 (local address 127.0.98.1:45239)
0927 02:34:03.556399 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:03.556411 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:03.557015 (+ 604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:03.557533 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:03.557536 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:03.557931 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:03.558521 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.558530 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.559770 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.559773 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.560725 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.560734 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.561453 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.561457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.561735 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.562402 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:03.562416 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:03.564627 (+ 2211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:34:03.566544 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:03.566550 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:03.566568 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:03.566872 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:03.567178 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:03.567181 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:03.567184 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:03.567341 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:03.567782 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:34:03.567789 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:34:03.568096 (+ 307us) client_negotiation.cc:769] Sending connection context
0927 02:34:03.568815 (+ 719us) client_negotiation.cc:240] Negotiation successful
0927 02:34:03.569194 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"mutex_wait_us":237,"spinlock_wait_cycles":10496,"thread_start_us":126,"threads_started":1}
I20260927 02:34:03.571511 7452 heartbeater.cc:347] Connected to a master server at 127.0.98.60:36279
I20260927 02:34:03.571528 7453 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46055
I20260927 02:34:03.571657 7459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.556555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45743 (local address 127.0.98.1:48291)
0927 02:34:03.556786 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:03.556799 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:03.557018 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:03.557494 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:03.557501 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:03.557925 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:03.558510 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.558524 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.559628 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.559631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.561582 (+ 1951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.561591 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.562612 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.562616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.562720 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.563152 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:03.563167 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:03.565367 (+ 2200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:34:03.567071 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:03.567074 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:03.567075 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:03.567277 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:03.567555 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:03.567557 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:03.567559 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:03.567631 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:03.568275 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:34:03.568279 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:34:03.568362 (+ 83us) client_negotiation.cc:769] Sending connection context
0927 02:34:03.569357 (+ 995us) client_negotiation.cc:240] Negotiation successful
0927 02:34:03.569473 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":141,"spinlock_wait_cycles":21888,"thread_start_us":67,"threads_started":1}
I20260927 02:34:03.571765 7453 heartbeater.cc:464] Registering TS with master...
I20260927 02:34:03.572469 7454 heartbeater.cc:347] Connected to a master server at 127.0.98.61:45743
I20260927 02:34:03.572502 7453 heartbeater.cc:511] Master 127.0.98.62:46055 requested a full tablet report, sending...
I20260927 02:34:03.572546 7454 heartbeater.cc:464] Registering TS with master...
I20260927 02:34:03.572700 7454 heartbeater.cc:511] Master 127.0.98.61:45743 requested a full tablet report, sending...
I20260927 02:34:03.571764 7452 heartbeater.cc:464] Registering TS with master...
I20260927 02:34:03.573130 7452 heartbeater.cc:511] Master 127.0.98.60:36279 requested a full tablet report, sending...
I20260927 02:34:03.573901 7081 ts_manager.cc:194] Registered new tserver with Master: 48c360f547f84f9881723197501c1cc0 (127.0.98.1:46237)
I20260927 02:34:03.574313 7273 ts_manager.cc:194] Registered new tserver with Master: 48c360f547f84f9881723197501c1cc0 (127.0.98.1:46237)
I20260927 02:34:03.575119 6489 ts_manager.cc:194] Re-registered known tserver with Master: 48c360f547f84f9881723197501c1cc0 (127.0.98.1:46237)
I20260927 02:34:03.575875 6489 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48967
W20260927 02:34:03.674623 7460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:34:03.674850 7460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:34:03.674888 7460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:34:03.678761 7460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:34:03.678807 7460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:34:03.678859 7460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:34:03.683833 7460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37325
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35353
--tserver_master_addrs=127.0.98.62:46055,127.0.98.61:45743,127.0.98.60:36279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:34:03.684687 7460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:34:03.685583 7460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:34:03.694334 7468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:03.694419 7460 server_base.cc:1034] running on GCE node
W20260927 02:34:03.694334 7466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:03.694339 7465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:03.694944 7460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:34:03.695498 7460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:34:03.696642 7460 hybrid_clock.cc:648] HybridClock initialized: now 1790476443696624 us; error 30 us; skew 500 ppm
Sep 27 02:34:03 dist-test-slave-grlk krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476443, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:34:03.699137 7460 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 02:34:03.699882 7460 webserver.cc:460] Webserver started at http://127.0.98.2:35353/ using document root <none> and password file <none>
I20260927 02:34:03.700405 7460 fs_manager.cc:359] Metadata directory not provided
I20260927 02:34:03.700477 7460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:34:03.703960 7460 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:34:03.705832 7475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:03.706949 7460 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:34:03.707053 7460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-1/wal
uuid: "733db7c7c5b54c6495ba5a80d1e50104"
format_stamp: "Formatted at 2026-09-27 02:34:02 on dist-test-slave-grlk"
server_key: "d20e81c4e1f17576a0e9b000a96a231f"
server_key_iv: "8199784bc714a7659b338f5bfc195046"
server_key_version: "encryptionkey@0"
I20260927 02:34:03.707535 7460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:34:03.731307 7460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:34:03.734319 7460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:34:03.734489 7460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:34:03.735052 7460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:34:03.735838 7460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:34:03.735883 7460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:03.735932 7460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:34:03.735960 7460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:03.744141 7460 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37325
I20260927 02:34:03.744180 7588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37325 every 8 connection(s)
I20260927 02:34:03.745134 7460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:34:03.754303 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 7460
I20260927 02:34:03.754482 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 6819
Sep 27 02:34:03 dist-test-slave-grlk krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476443, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 02:34:03 dist-test-slave-grlk krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476443, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 27 02:34:03 dist-test-slave-grlk krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476443, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 02:34:03.760281 7456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.746939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34407 (local address 127.0.98.60:36279)
0927 02:34:03.747055 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:03.747059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:03.747760 (+ 701us) server_negotiation.cc:407] Connection header received
0927 02:34:03.749143 (+ 1383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:03.749146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:03.749210 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:03.749275 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:03.750532 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.751351 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.752755 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.753139 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.753212 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.756259 (+ 3047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:03.756263 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:03.756265 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:03.756289 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:03.757772 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.758561 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.758564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.758566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.758653 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.759025 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.759028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.759030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.759241 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:03.759310 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:03.760001 (+ 691us) server_negotiation.cc:299] Negotiation successful
0927 02:34:03.760122 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 02:34:03.761667 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43921
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:46055
--webserver_port=38455
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:46055,127.0.98.61:45743
--tserver_master_addrs=127.0.98.62:46055,127.0.98.61:45743,127.0.98.60:36279 with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:34:03.762080 7593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.747161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36279 (local address 127.0.98.2:34407)
0927 02:34:03.747603 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:03.747649 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:03.748924 (+ 1275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:03.749440 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:03.749447 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:03.749855 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:03.750411 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.750426 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.751822 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.751825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.752634 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.752643 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.753403 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.753407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.753508 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.753865 (+ 357us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:03.753880 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:03.756101 (+ 2221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:34:03.757885 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:03.757891 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:03.757905 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:03.758424 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:03.758763 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:03.758765 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:03.758767 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:03.758920 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:03.759383 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:34:03.759389 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:34:03.759715 (+ 326us) client_negotiation.cc:769] Sending connection context
0927 02:34:03.760629 (+ 914us) client_negotiation.cc:240] Negotiation successful
0927 02:34:03.760959 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":5,"thread_start_us":109,"threads_started":1}
I20260927 02:34:03.762183 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.746790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33375 (local address 127.0.98.62:46055)
0927 02:34:03.746932 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:03.746937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:03.748060 (+ 1123us) server_negotiation.cc:407] Connection header received
0927 02:34:03.749227 (+ 1167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:03.749231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:03.749280 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:03.749349 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:03.750587 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.751348 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.752412 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.752619 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.752700 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.758401 (+ 5701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:03.758404 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:03.758407 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:03.758431 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:03.760278 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.760985 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.760989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.760991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.761069 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.761349 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.761352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.761354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.761546 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:03.761618 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:03.761925 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 02:34:03.762028 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 02:34:03.762214 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.748442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49015 (local address 127.0.98.61:45743)
0927 02:34:03.748563 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:03.748566 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:03.748580 (+ 14us) server_negotiation.cc:407] Connection header received
0927 02:34:03.749171 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:03.749175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:03.749230 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:03.749285 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:03.750571 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.751092 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.752606 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.752921 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.753025 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.758587 (+ 5562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:03.758590 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:03.758592 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:03.758616 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:03.760087 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.760849 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.760852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.760854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.760967 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.761245 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.761249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.761251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.761447 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:03.761513 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:03.761977 (+ 464us) server_negotiation.cc:299] Negotiation successful
0927 02:34:03.762084 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 02:34:03.762910 7594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.747158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.98.2:33375)
0927 02:34:03.747961 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:03.747975 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:03.748948 (+ 973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:03.749473 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:03.749476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:03.749886 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:03.750425 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.750432 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.751467 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.751471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.752306 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.752314 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.752740 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.752743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.752900 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.753425 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:03.753441 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:03.758259 (+ 4818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:34:03.760401 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:03.760404 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:03.760405 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:03.760852 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:03.761175 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:03.761178 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:03.761180 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:03.761255 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:03.761673 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:34:03.761676 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:34:03.761761 (+ 85us) client_negotiation.cc:769] Sending connection context
0927 02:34:03.762295 (+ 534us) client_negotiation.cc:240] Negotiation successful
0927 02:34:03.762422 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"mutex_wait_us":198,"spinlock_wait_cycles":39808,"thread_start_us":564,"threads_started":1}
I20260927 02:34:03.763305 7589 heartbeater.cc:347] Connected to a master server at 127.0.98.60:36279
I20260927 02:34:03.763473 7595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.748092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45743 (local address 127.0.98.2:49015)
0927 02:34:03.748322 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:03.748336 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:03.748954 (+ 618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:03.749442 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:03.749450 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:03.749893 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:03.750422 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.750432 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.751238 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.751240 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.752464 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.752474 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.753182 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.753186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.753277 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.753709 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:03.753722 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:03.758443 (+ 4721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:34:03.760193 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:03.760196 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:03.760198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:03.760745 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:03.761073 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:03.761076 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:03.761078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:03.761156 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:03.761581 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:34:03.761584 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:34:03.761665 (+ 81us) client_negotiation.cc:769] Sending connection context
0927 02:34:03.762812 (+ 1147us) client_negotiation.cc:240] Negotiation successful
0927 02:34:03.763236 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":80,"spinlock_wait_cycles":29568,"thread_start_us":70,"threads_started":1}
I20260927 02:34:03.763547 7589 heartbeater.cc:464] Registering TS with master...
I20260927 02:34:03.763687 7590 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46055
I20260927 02:34:03.763944 7590 heartbeater.cc:464] Registering TS with master...
I20260927 02:34:03.764333 7591 heartbeater.cc:347] Connected to a master server at 127.0.98.61:45743
I20260927 02:34:03.764411 7591 heartbeater.cc:464] Registering TS with master...
I20260927 02:34:03.764423 7589 heartbeater.cc:511] Master 127.0.98.60:36279 requested a full tablet report, sending...
I20260927 02:34:03.764423 7590 heartbeater.cc:511] Master 127.0.98.62:46055 requested a full tablet report, sending...
I20260927 02:34:03.764570 7591 heartbeater.cc:511] Master 127.0.98.61:45743 requested a full tablet report, sending...
I20260927 02:34:03.765481 6489 ts_manager.cc:194] Re-registered known tserver with Master: 733db7c7c5b54c6495ba5a80d1e50104 (127.0.98.2:37325)
I20260927 02:34:03.765555 7273 ts_manager.cc:194] Registered new tserver with Master: 733db7c7c5b54c6495ba5a80d1e50104 (127.0.98.2:37325)
I20260927 02:34:03.766129 6489 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33375
I20260927 02:34:03.766590 7081 ts_manager.cc:194] Registered new tserver with Master: 733db7c7c5b54c6495ba5a80d1e50104 (127.0.98.2:37325)
W20260927 02:34:03.869046 7596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:34:03.869338 7596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:34:03.869381 7596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:34:03.873454 7596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:34:03.873510 7596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:34:03.873566 7596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:34:03.878595 7596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43921
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38455
--tserver_master_addrs=127.0.98.62:46055,127.0.98.61:45743,127.0.98.60:36279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:34:03.879494 7596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:34:03.880402 7596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:34:03.889456 7602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:03.889477 7596 server_base.cc:1034] running on GCE node
W20260927 02:34:03.889560 7604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:03.889688 7601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:03.890030 7596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:34:03.890578 7596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:34:03.891742 7596 hybrid_clock.cc:648] HybridClock initialized: now 1790476443891730 us; error 24 us; skew 500 ppm
Sep 27 02:34:03 dist-test-slave-grlk krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476443, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:34:03.894168 7596 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 02:34:03.894881 7596 webserver.cc:460] Webserver started at http://127.0.98.3:38455/ using document root <none> and password file <none>
I20260927 02:34:03.895358 7596 fs_manager.cc:359] Metadata directory not provided
I20260927 02:34:03.895424 7596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:34:03.898870 7596 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:34:03.900673 7611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:03.901782 7596 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:34:03.901877 7596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-2/wal
uuid: "65d9be8a776a46f6a53142ae0accd7ee"
format_stamp: "Formatted at 2026-09-27 02:34:02 on dist-test-slave-grlk"
server_key: "041d5bad0d12127ed86c06ed62af722f"
server_key_iv: "fbe1a0a96e3fab7592824ce6985bc719"
server_key_version: "encryptionkey@0"
I20260927 02:34:03.902392 7596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:34:03.931092 7616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.927464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36279 (local address 127.0.0.1:34400)
0927 02:34:03.927676 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:03.927689 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:03.927782 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:03.928051 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:03.928053 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:03.928069 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:34:03.928178 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.928183 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.929132 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.929135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.929935 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.929944 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.930761 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.930764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.930836 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.930858 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 02:34:03.930916 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 02:34:03.930975 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":59,"threads_started":1}
I20260927 02:34:03.931109 7456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.927548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34400 (local address 127.0.98.60:36279)
0927 02:34:03.927678 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:03.927682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:03.927749 (+ 67us) server_negotiation.cc:407] Connection header received
0927 02:34:03.927867 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:03.927870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:03.927926 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:03.928026 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:34:03.928264 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.929043 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.930038 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.930638 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.930730 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.930859 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:03.930931 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 02:34:03.930992 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 02:34:03.931864 7283 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260927 02:34:03.932260 7312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aff7cb041f714927ae5a16a8dc279205. Latest consensus state: current_term: 1 leader_uuid: "aff7cb041f714927ae5a16a8dc279205" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } peers { permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45743 } attrs { promote: false } } peers { permanent_uuid: "11b7f1d9d8944fd19085f2cc3c659137" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36279 } attrs { promote: true } } }
I20260927 02:34:03.932415 7312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137 [sys.catalog]: This master's current role is: LEARNER
I20260927 02:34:03.934362 7596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:34:03.937188 7596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:34:03.937336 7596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:34:03.937839 7596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:34:03.938606 7596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:34:03.938656 7596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:03.938707 7596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:34:03.938731 7596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:03.946612 7596 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43921
I20260927 02:34:03.946646 7725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43921 every 8 connection(s)
I20260927 02:34:03.947482 7596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:34:03.948710 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 7596
Sep 27 02:34:03 dist-test-slave-grlk krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476443, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 02:34:03 dist-test-slave-grlk krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476443, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 02:34:03.960994 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.950940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57756 (local address 127.0.98.62:46055)
0927 02:34:03.951040 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:03.951044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:03.951674 (+ 630us) server_negotiation.cc:407] Connection header received
0927 02:34:03.951813 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:03.951816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:03.951858 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:03.951900 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:03.952830 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.953613 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.954784 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.955113 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.955179 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.956832 (+ 1653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:03.956835 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:03.956837 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:03.956857 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:03.958412 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.959766 (+ 1354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.959770 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.959771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.959852 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.960109 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.960112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.960114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.960301 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:03.960391 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:03.960703 (+ 312us) server_negotiation.cc:299] Negotiation successful
0927 02:34:03.960810 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":40}
I20260927 02:34:03.964705 7456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.949073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37837 (local address 127.0.98.60:36279)
0927 02:34:03.949178 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:03.949182 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:03.950661 (+ 1479us) server_negotiation.cc:407] Connection header received
0927 02:34:03.951766 (+ 1105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:03.951770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:03.951819 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:03.951866 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:03.953188 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.953934 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.955647 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.955985 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.956069 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.959905 (+ 3836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:03.959909 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:03.959911 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:03.959935 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:03.962486 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.963059 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.963062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.963064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.963143 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.963509 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.963512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.963514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.963717 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:03.963797 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:03.964423 (+ 626us) server_negotiation.cc:299] Negotiation successful
0927 02:34:03.964542 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 02:34:03.965310 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.951000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41947 (local address 127.0.98.62:46055)
0927 02:34:03.951203 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:03.951206 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:03.951444 (+ 238us) server_negotiation.cc:407] Connection header received
0927 02:34:03.951753 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:03.951756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:03.951808 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:03.951889 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:03.953195 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.953896 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.955665 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.955988 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.956069 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.959304 (+ 3235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:03.959307 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:03.959310 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:03.959333 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:03.961968 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.963058 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.963061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.963062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.963360 (+ 298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.963629 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.963632 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.963634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.963866 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:03.964155 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:03.965031 (+ 876us) server_negotiation.cc:299] Negotiation successful
0927 02:34:03.965149 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":90,"threads_started":1}
Sep 27 02:34:03 dist-test-slave-grlk krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476443, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 02:34:03.966387 7734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.949603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.98.3:41947)
0927 02:34:03.951356 (+ 1753us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:03.951369 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:03.951575 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:03.952042 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:03.952048 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:03.952463 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:03.953035 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.953055 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.954031 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.954034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.955548 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.955558 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.956204 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.956209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.956467 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.957141 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:03.957153 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:03.959160 (+ 2007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:34:03.962084 (+ 2924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:03.962089 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:03.962103 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:03.962953 (+ 850us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:03.963456 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:03.963459 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:03.963461 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:03.963538 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:03.964210 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:34:03.964213 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:34:03.964502 (+ 289us) client_negotiation.cc:769] Sending connection context
0927 02:34:03.964613 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 02:34:03.965767 (+ 1154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":175,"spinlock_wait_cycles":1408,"thread_start_us":631,"threads_started":1}
I20260927 02:34:03.966639 7735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.949603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36279 (local address 127.0.98.3:37837)
0927 02:34:03.950523 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:03.950574 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:03.951569 (+ 995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:03.952043 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:03.952050 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:03.952474 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:03.953036 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.953055 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.954062 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.954065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.955551 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.955561 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.956204 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.956209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.956463 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.957103 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:03.957123 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:03.959765 (+ 2642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:34:03.962594 (+ 2829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:03.962597 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:03.962599 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:03.962966 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:03.963246 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:03.963248 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:03.963250 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:03.963413 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:03.963858 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:34:03.963863 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:34:03.964150 (+ 287us) client_negotiation.cc:769] Sending connection context
0927 02:34:03.964833 (+ 683us) client_negotiation.cc:240] Negotiation successful
0927 02:34:03.965341 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"mutex_wait_us":254,"spinlock_wait_cycles":43008,"thread_start_us":133,"threads_started":1}
I20260927 02:34:03.963640 6489 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57756:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 02:34:03.967813 7726 heartbeater.cc:347] Connected to a master server at 127.0.98.60:36279
I20260927 02:34:03.968030 7726 heartbeater.cc:464] Registering TS with master...
W20260927 02:34:03.968166 6489 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:34:03.968494 7726 heartbeater.cc:511] Master 127.0.98.60:36279 requested a full tablet report, sending...
I20260927 02:34:03.969038 7727 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46055
I20260927 02:34:03.969129 7727 heartbeater.cc:464] Registering TS with master...
I20260927 02:34:03.969293 7727 heartbeater.cc:511] Master 127.0.98.62:46055 requested a full tablet report, sending...
I20260927 02:34:03.970109 6488 ts_manager.cc:194] Re-registered known tserver with Master: 65d9be8a776a46f6a53142ae0accd7ee (127.0.98.3:43921)
I20260927 02:34:03.970372 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.950768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59879 (local address 127.0.98.61:45743)
0927 02:34:03.951234 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:03.951237 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:03.957826 (+ 6589us) server_negotiation.cc:407] Connection header received
0927 02:34:03.957979 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:03.957982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:03.958027 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:03.958084 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:03.958607 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.959361 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.963182 (+ 3821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.963477 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.963547 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.966685 (+ 3138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:03.966689 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:03.966692 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:03.966714 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:03.968199 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.968684 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.968687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.968689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.968906 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.969386 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.969389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.969391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.969593 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:03.969718 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:03.970052 (+ 334us) server_negotiation.cc:299] Negotiation successful
0927 02:34:03.970223 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":68,"threads_started":1}
I20260927 02:34:03.970788 6488 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41947
I20260927 02:34:03.971138 7737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.950612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45743 (local address 127.0.98.3:59879)
0927 02:34:03.957751 (+ 7139us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:03.957764 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:03.957885 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:03.958226 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:03.958229 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:03.958384 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:03.958499 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.958506 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.961118 (+ 2612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.961121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.961866 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.961876 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.964013 (+ 2137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.964016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.964101 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.965121 (+ 1020us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:03.965133 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:03.966536 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:34:03.968312 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:03.968315 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:03.968316 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:03.968538 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:03.969020 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:03.969022 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:03.969024 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:03.969239 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:03.969769 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:34:03.969771 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:34:03.969854 (+ 83us) client_negotiation.cc:769] Sending connection context
0927 02:34:03.970737 (+ 883us) client_negotiation.cc:240] Negotiation successful
0927 02:34:03.970948 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7078,"thread_start_us":76,"threads_started":1}
I20260927 02:34:03.971585 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.951207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34410 (local address 127.0.98.60:36279)
0927 02:34:03.951407 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:03.951417 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:03.956136 (+ 4719us) server_negotiation.cc:407] Connection header received
0927 02:34:03.956173 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:03.956176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:03.956218 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:03.956294 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:03.958526 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.959303 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.960320 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.960622 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.960703 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.962873 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:03.962876 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:03.962879 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:03.962900 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:03.964870 (+ 1970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.969993 (+ 5123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.969996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.969998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.970080 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.970362 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.970365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.970367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.970576 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:03.970706 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:03.971313 (+ 607us) server_negotiation.cc:299] Negotiation successful
0927 02:34:03.971426 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":82,"threads_started":1}
I20260927 02:34:03.973273 7728 heartbeater.cc:347] Connected to a master server at 127.0.98.61:45743
I20260927 02:34:03.973351 7728 heartbeater.cc:464] Registering TS with master...
I20260927 02:34:03.973269 7048 sys_catalog.cc:455] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aff7cb041f714927ae5a16a8dc279205" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } peers { permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45743 } attrs { promote: false } } peers { permanent_uuid: "11b7f1d9d8944fd19085f2cc3c659137" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36279 } attrs { promote: true } } }
I20260927 02:34:03.973390 7048 sys_catalog.cc:458] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205 [sys.catalog]: This master's current role is: LEADER
I20260927 02:34:03.973874 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.950708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60830 (local address 127.0.98.61:45743)
0927 02:34:03.950937 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:03.950941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:03.953055 (+ 2114us) server_negotiation.cc:407] Connection header received
0927 02:34:03.953222 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:03.953225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:03.953272 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:03.953319 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:03.954154 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.954894 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.961056 (+ 6162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.961364 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.961435 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.970049 (+ 8614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:03.970053 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:03.970055 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:03.970078 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:03.971771 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.972212 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.972216 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.972217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.972300 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:03.972607 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:03.972610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:03.972612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:03.972820 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:03.972967 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:03.973598 (+ 631us) server_negotiation.cc:299] Negotiation successful
0927 02:34:03.973735 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165}
I20260927 02:34:03.973876 7728 heartbeater.cc:511] Master 127.0.98.61:45743 requested a full tablet report, sending...
I20260927 02:34:03.977339 7273 ts_manager.cc:194] Registered new tserver with Master: 65d9be8a776a46f6a53142ae0accd7ee (127.0.98.3:43921)
I20260927 02:34:03.978180 7081 ts_manager.cc:194] Registered new tserver with Master: 65d9be8a776a46f6a53142ae0accd7ee (127.0.98.3:43921)
I20260927 02:34:03.979409 7048 raft_consensus.cc:1050] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205: attempting to promote NON_VOTER 11b7f1d9d8944fd19085f2cc3c659137 to VOTER
I20260927 02:34:03.979733 7048 consensus_queue.cc:237] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } peers { permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45743 } attrs { promote: false } } peers { permanent_uuid: "11b7f1d9d8944fd19085f2cc3c659137" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36279 } attrs { promote: false } }
I20260927 02:34:03.980782 7098 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [term 1 FOLLOWER]: Refusing update from remote peer aff7cb041f714927ae5a16a8dc279205: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 02:34:03.981141 7048 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45743 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 02:34:03.981909 7282 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137 [term 1 LEARNER]: Refusing update from remote peer aff7cb041f714927ae5a16a8dc279205: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 02:34:03.982736 7751 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11b7f1d9d8944fd19085f2cc3c659137" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 02:34:03.983538 7098 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 11b7f1d9d8944fd19085f2cc3c659137 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } peers { permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45743 } attrs { promote: false } } peers { permanent_uuid: "11b7f1d9d8944fd19085f2cc3c659137" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36279 } attrs { promote: false } } }
I20260927 02:34:03.983180 7751 raft_consensus.cc:2932] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 11b7f1d9d8944fd19085f2cc3c659137 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } peers { permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45743 } attrs { promote: false } } peers { permanent_uuid: "11b7f1d9d8944fd19085f2cc3c659137" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36279 } attrs { promote: false } } }
I20260927 02:34:03.985055 7753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aff7cb041f714927ae5a16a8dc279205" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } peers { permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45743 } attrs { promote: false } } peers { permanent_uuid: "11b7f1d9d8944fd19085f2cc3c659137" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36279 } attrs { promote: false } } }
I20260927 02:34:03.985235 7753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:34:03.985692 7282 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 11b7f1d9d8944fd19085f2cc3c659137 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } peers { permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45743 } attrs { promote: false } } peers { permanent_uuid: "11b7f1d9d8944fd19085f2cc3c659137" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36279 } attrs { promote: false } } }
I20260927 02:34:03.986001 7048 sys_catalog.cc:455] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "aff7cb041f714927ae5a16a8dc279205" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } peers { permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45743 } attrs { promote: false } } peers { permanent_uuid: "11b7f1d9d8944fd19085f2cc3c659137" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36279 } attrs { promote: false } } }
I20260927 02:34:03.986092 7048 sys_catalog.cc:458] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205 [sys.catalog]: This master's current role is: LEADER
I20260927 02:34:03.986033 7744 sys_catalog.cc:455] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aff7cb041f714927ae5a16a8dc279205" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } peers { permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45743 } attrs { promote: false } } peers { permanent_uuid: "11b7f1d9d8944fd19085f2cc3c659137" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36279 } attrs { promote: false } } }
I20260927 02:34:03.986130 7744 sys_catalog.cc:458] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205 [sys.catalog]: This master's current role is: LEADER
I20260927 02:34:03.986284 7744 sys_catalog.cc:455] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aff7cb041f714927ae5a16a8dc279205" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } peers { permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45743 } attrs { promote: false } } peers { permanent_uuid: "11b7f1d9d8944fd19085f2cc3c659137" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36279 } attrs { promote: false } } }
I20260927 02:34:03.986354 7744 sys_catalog.cc:458] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205 [sys.catalog]: This master's current role is: LEADER
I20260927 02:34:03.987469 7312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aff7cb041f714927ae5a16a8dc279205" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } peers { permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45743 } attrs { promote: false } } peers { permanent_uuid: "11b7f1d9d8944fd19085f2cc3c659137" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36279 } attrs { promote: false } } }
I20260927 02:34:03.987586 7312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:34:03.992632 7616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.987094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46237 (local address 127.0.0.1:36714)
0927 02:34:03.987251 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:03.987265 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:03.987343 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:03.988704 (+ 1361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:03.988707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:03.988725 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:34:03.988825 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.988831 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.990608 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.990611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.991313 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.991319 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.992269 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.992273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.992378 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.992402 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 02:34:03.992444 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 02:34:03.992506 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":3}
I20260927 02:34:03.993088 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.987696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43921 (local address 127.0.0.1:43010)
0927 02:34:03.987926 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:03.987939 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:03.988019 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:03.988827 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:03.988829 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:03.988846 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:34:03.988966 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.988973 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.990076 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.990079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.991137 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.991147 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.992202 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.992206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.992765 (+ 559us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.992787 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 02:34:03.992845 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 02:34:03.992906 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":463,"thread_start_us":91,"threads_started":1}
I20260927 02:34:03.993172 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.988139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43010 (local address 127.0.98.3:43921)
0927 02:34:03.988457 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:03.988465 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:03.988481 (+ 16us) server_negotiation.cc:407] Connection header received
0927 02:34:03.988550 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:03.988554 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:03.988715 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:03.988859 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:34:03.989060 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.989968 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.991265 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.992087 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.992318 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.992430 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:03.992879 (+ 449us) server_negotiation.cc:299] Negotiation successful
0927 02:34:03.993003 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":140,"threads_started":1}
I20260927 02:34:03.993629 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.987250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36714 (local address 127.0.98.1:46237)
0927 02:34:03.987672 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:03.987684 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:03.987705 (+ 21us) server_negotiation.cc:407] Connection header received
0927 02:34:03.987784 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:03.987789 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:03.988030 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:03.989254 (+ 1224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:34:03.989486 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.990521 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.991415 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.992145 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.993032 (+ 887us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.993411 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:03.993452 (+ 41us) server_negotiation.cc:299] Negotiation successful
0927 02:34:03.993515 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":108,"threads_started":1}
I20260927 02:34:03.994916 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.988710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37325 (local address 127.0.0.1:42574)
0927 02:34:03.989410 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:03.989422 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:03.989502 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:03.990229 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:03.990232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:03.990248 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:34:03.990333 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.990338 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.991710 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.991713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.992768 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:03.992778 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.994562 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.994566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:03.994660 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.994681 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 02:34:03.994744 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 02:34:03.994804 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":73,"threads_started":1}
I20260927 02:34:03.995361 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:03.989081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42574 (local address 127.0.98.2:37325)
0927 02:34:03.989722 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:03.989733 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:03.989753 (+ 20us) server_negotiation.cc:407] Connection header received
0927 02:34:03.989819 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:03.989823 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:03.989983 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:03.990111 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:34:03.990448 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.991491 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.992891 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:03.994270 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:03.994375 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:03.994502 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:03.995145 (+ 643us) server_negotiation.cc:299] Negotiation successful
0927 02:34:03.995235 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":86,"threads_started":1}
I20260927 02:34:03.995939 7386 tablet_service.cc:1467] Processing CreateTablet for tablet 72d4ea321b2f45bca9ed03e52012e9c4 (DEFAULT_TABLE table=test-table [id=f97b1d9508de4e438eef028d349882a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:34:03.995952 7660 tablet_service.cc:1467] Processing CreateTablet for tablet 72d4ea321b2f45bca9ed03e52012e9c4 (DEFAULT_TABLE table=test-table [id=f97b1d9508de4e438eef028d349882a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:34:03.996799 7386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72d4ea321b2f45bca9ed03e52012e9c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:34:03.996922 7660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72d4ea321b2f45bca9ed03e52012e9c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:34:03.997692 7523 tablet_service.cc:1467] Processing CreateTablet for tablet 72d4ea321b2f45bca9ed03e52012e9c4 (DEFAULT_TABLE table=test-table [id=f97b1d9508de4e438eef028d349882a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:34:03.998510 7523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72d4ea321b2f45bca9ed03e52012e9c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:34:04.003217 7759 tablet_bootstrap.cc:492] T 72d4ea321b2f45bca9ed03e52012e9c4 P 48c360f547f84f9881723197501c1cc0: Bootstrap starting.
I20260927 02:34:04.005851 7759 tablet_bootstrap.cc:654] T 72d4ea321b2f45bca9ed03e52012e9c4 P 48c360f547f84f9881723197501c1cc0: Neither blocks nor log segments found. Creating new log.
I20260927 02:34:04.006618 7759 log.cc:824] T 72d4ea321b2f45bca9ed03e52012e9c4 P 48c360f547f84f9881723197501c1cc0: Log is configured to *not* fsync() on all Append() calls
I20260927 02:34:04.008527 7759 tablet_bootstrap.cc:492] T 72d4ea321b2f45bca9ed03e52012e9c4 P 48c360f547f84f9881723197501c1cc0: No bootstrap required, opened a new log
I20260927 02:34:04.008690 7759 ts_tablet_manager.cc:1397] T 72d4ea321b2f45bca9ed03e52012e9c4 P 48c360f547f84f9881723197501c1cc0: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260927 02:34:04.009814 7760 tablet_bootstrap.cc:492] T 72d4ea321b2f45bca9ed03e52012e9c4 P 733db7c7c5b54c6495ba5a80d1e50104: Bootstrap starting.
I20260927 02:34:04.012164 7760 tablet_bootstrap.cc:654] T 72d4ea321b2f45bca9ed03e52012e9c4 P 733db7c7c5b54c6495ba5a80d1e50104: Neither blocks nor log segments found. Creating new log.
I20260927 02:34:04.012988 7760 log.cc:824] T 72d4ea321b2f45bca9ed03e52012e9c4 P 733db7c7c5b54c6495ba5a80d1e50104: Log is configured to *not* fsync() on all Append() calls
I20260927 02:34:04.012915 7759 raft_consensus.cc:348] T 72d4ea321b2f45bca9ed03e52012e9c4 P 48c360f547f84f9881723197501c1cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "733db7c7c5b54c6495ba5a80d1e50104" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37325 } } peers { permanent_uuid: "48c360f547f84f9881723197501c1cc0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46237 } } peers { permanent_uuid: "65d9be8a776a46f6a53142ae0accd7ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43921 } }
I20260927 02:34:04.013180 7759 raft_consensus.cc:374] T 72d4ea321b2f45bca9ed03e52012e9c4 P 48c360f547f84f9881723197501c1cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:34:04.013226 7759 raft_consensus.cc:729] T 72d4ea321b2f45bca9ed03e52012e9c4 P 48c360f547f84f9881723197501c1cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48c360f547f84f9881723197501c1cc0, State: Initialized, Role: FOLLOWER
I20260927 02:34:04.013644 7759 consensus_queue.cc:260] T 72d4ea321b2f45bca9ed03e52012e9c4 P 48c360f547f84f9881723197501c1cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "733db7c7c5b54c6495ba5a80d1e50104" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37325 } } peers { permanent_uuid: "48c360f547f84f9881723197501c1cc0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46237 } } peers { permanent_uuid: "65d9be8a776a46f6a53142ae0accd7ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43921 } }
I20260927 02:34:04.014510 7759 ts_tablet_manager.cc:1428] T 72d4ea321b2f45bca9ed03e52012e9c4 P 48c360f547f84f9881723197501c1cc0: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260927 02:34:04.014822 7453 heartbeater.cc:503] Master 127.0.98.62:46055 was elected leader, sending a full tablet report...
I20260927 02:34:04.015515 7763 tablet_bootstrap.cc:492] T 72d4ea321b2f45bca9ed03e52012e9c4 P 65d9be8a776a46f6a53142ae0accd7ee: Bootstrap starting.
I20260927 02:34:04.017472 7763 tablet_bootstrap.cc:654] T 72d4ea321b2f45bca9ed03e52012e9c4 P 65d9be8a776a46f6a53142ae0accd7ee: Neither blocks nor log segments found. Creating new log.
I20260927 02:34:04.018740 7763 log.cc:824] T 72d4ea321b2f45bca9ed03e52012e9c4 P 65d9be8a776a46f6a53142ae0accd7ee: Log is configured to *not* fsync() on all Append() calls
I20260927 02:34:04.019351 7760 tablet_bootstrap.cc:492] T 72d4ea321b2f45bca9ed03e52012e9c4 P 733db7c7c5b54c6495ba5a80d1e50104: No bootstrap required, opened a new log
I20260927 02:34:04.019533 7760 ts_tablet_manager.cc:1397] T 72d4ea321b2f45bca9ed03e52012e9c4 P 733db7c7c5b54c6495ba5a80d1e50104: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260927 02:34:04.020351 7763 tablet_bootstrap.cc:492] T 72d4ea321b2f45bca9ed03e52012e9c4 P 65d9be8a776a46f6a53142ae0accd7ee: No bootstrap required, opened a new log
I20260927 02:34:04.020504 7763 ts_tablet_manager.cc:1397] T 72d4ea321b2f45bca9ed03e52012e9c4 P 65d9be8a776a46f6a53142ae0accd7ee: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 02:34:04.022724 7760 raft_consensus.cc:348] T 72d4ea321b2f45bca9ed03e52012e9c4 P 733db7c7c5b54c6495ba5a80d1e50104 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "733db7c7c5b54c6495ba5a80d1e50104" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37325 } } peers { permanent_uuid: "48c360f547f84f9881723197501c1cc0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46237 } } peers { permanent_uuid: "65d9be8a776a46f6a53142ae0accd7ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43921 } }
I20260927 02:34:04.022908 7760 raft_consensus.cc:374] T 72d4ea321b2f45bca9ed03e52012e9c4 P 733db7c7c5b54c6495ba5a80d1e50104 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:34:04.022940 7760 raft_consensus.cc:729] T 72d4ea321b2f45bca9ed03e52012e9c4 P 733db7c7c5b54c6495ba5a80d1e50104 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 733db7c7c5b54c6495ba5a80d1e50104, State: Initialized, Role: FOLLOWER
I20260927 02:34:04.023308 7760 consensus_queue.cc:260] T 72d4ea321b2f45bca9ed03e52012e9c4 P 733db7c7c5b54c6495ba5a80d1e50104 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "733db7c7c5b54c6495ba5a80d1e50104" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37325 } } peers { permanent_uuid: "48c360f547f84f9881723197501c1cc0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46237 } } peers { permanent_uuid: "65d9be8a776a46f6a53142ae0accd7ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43921 } }
I20260927 02:34:04.023566 7763 raft_consensus.cc:348] T 72d4ea321b2f45bca9ed03e52012e9c4 P 65d9be8a776a46f6a53142ae0accd7ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "733db7c7c5b54c6495ba5a80d1e50104" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37325 } } peers { permanent_uuid: "48c360f547f84f9881723197501c1cc0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46237 } } peers { permanent_uuid: "65d9be8a776a46f6a53142ae0accd7ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43921 } }
I20260927 02:34:04.023743 7763 raft_consensus.cc:374] T 72d4ea321b2f45bca9ed03e52012e9c4 P 65d9be8a776a46f6a53142ae0accd7ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:34:04.023790 7763 raft_consensus.cc:729] T 72d4ea321b2f45bca9ed03e52012e9c4 P 65d9be8a776a46f6a53142ae0accd7ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65d9be8a776a46f6a53142ae0accd7ee, State: Initialized, Role: FOLLOWER
I20260927 02:34:04.023859 7590 heartbeater.cc:503] Master 127.0.98.62:46055 was elected leader, sending a full tablet report...
I20260927 02:34:04.024145 7763 consensus_queue.cc:260] T 72d4ea321b2f45bca9ed03e52012e9c4 P 65d9be8a776a46f6a53142ae0accd7ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "733db7c7c5b54c6495ba5a80d1e50104" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37325 } } peers { permanent_uuid: "48c360f547f84f9881723197501c1cc0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46237 } } peers { permanent_uuid: "65d9be8a776a46f6a53142ae0accd7ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43921 } }
I20260927 02:34:04.025043 7763 ts_tablet_manager.cc:1428] T 72d4ea321b2f45bca9ed03e52012e9c4 P 65d9be8a776a46f6a53142ae0accd7ee: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20260927 02:34:04.025199 7727 heartbeater.cc:503] Master 127.0.98.62:46055 was elected leader, sending a full tablet report...
I20260927 02:34:04.027046 7760 ts_tablet_manager.cc:1428] T 72d4ea321b2f45bca9ed03e52012e9c4 P 733db7c7c5b54c6495ba5a80d1e50104: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
W20260927 02:34:04.055037 7455 tablet.cc:2369] T 72d4ea321b2f45bca9ed03e52012e9c4 P 48c360f547f84f9881723197501c1cc0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:34:04.198629 7729 tablet.cc:2369] T 72d4ea321b2f45bca9ed03e52012e9c4 P 65d9be8a776a46f6a53142ae0accd7ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:34:04.242466 7767 raft_consensus.cc:482] T 72d4ea321b2f45bca9ed03e52012e9c4 P 65d9be8a776a46f6a53142ae0accd7ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:34:04.242677 7767 raft_consensus.cc:504] T 72d4ea321b2f45bca9ed03e52012e9c4 P 65d9be8a776a46f6a53142ae0accd7ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "733db7c7c5b54c6495ba5a80d1e50104" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37325 } } peers { permanent_uuid: "48c360f547f84f9881723197501c1cc0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46237 } } peers { permanent_uuid: "65d9be8a776a46f6a53142ae0accd7ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43921 } }
I20260927 02:34:04.243839 7767 leader_election.cc:288] T 72d4ea321b2f45bca9ed03e52012e9c4 P 65d9be8a776a46f6a53142ae0accd7ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 733db7c7c5b54c6495ba5a80d1e50104 (127.0.98.2:37325), 48c360f547f84f9881723197501c1cc0 (127.0.98.1:46237)
W20260927 02:34:04.246487 7592 tablet.cc:2369] T 72d4ea321b2f45bca9ed03e52012e9c4 P 733db7c7c5b54c6495ba5a80d1e50104: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:34:04.247505 7734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:04.244054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46237 (local address 127.0.98.3:38291)
0927 02:34:04.244296 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:04.244311 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:04.244411 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:04.244682 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:04.244684 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:04.244698 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:34:04.244796 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:04.244804 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:04.245542 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:04.245545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:04.246341 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:04.246350 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:04.247188 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:04.247191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:04.247265 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:04.247285 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 02:34:04.247344 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 02:34:04.247392 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":78}
I20260927 02:34:04.247599 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:04.244285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38291 (local address 127.0.98.1:46237)
0927 02:34:04.244417 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:04.244421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:04.244433 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:34:04.244539 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:04.244542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:04.244589 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:04.244675 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:34:04.244909 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:04.245435 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:04.246489 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:04.247070 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:04.247183 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:04.247249 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:04.247379 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 02:34:04.247464 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 02:34:04.247740 7737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:04.244054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37325 (local address 127.0.98.3:35569)
0927 02:34:04.244158 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:04.244172 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:04.244263 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:04.244549 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:04.244552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:04.244568 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:34:04.244689 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:04.244696 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:04.245506 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:04.245509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:04.246334 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:04.246344 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:04.247324 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:04.247329 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:04.247431 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:04.247456 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 02:34:04.247538 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 02:34:04.247603 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":24}
I20260927 02:34:04.247826 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:04.244191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35569 (local address 127.0.98.2:37325)
0927 02:34:04.244314 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:04.244319 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:04.244335 (+ 16us) server_negotiation.cc:407] Connection header received
0927 02:34:04.244394 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:04.244397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:04.244449 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:04.244533 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:34:04.244797 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:04.245399 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:04.246462 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:04.247070 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:04.247183 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:04.247249 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:04.247574 (+ 325us) server_negotiation.cc:299] Negotiation successful
0927 02:34:04.247724 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 02:34:04.248322 7543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72d4ea321b2f45bca9ed03e52012e9c4" candidate_uuid: "65d9be8a776a46f6a53142ae0accd7ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "733db7c7c5b54c6495ba5a80d1e50104" is_pre_election: true
I20260927 02:34:04.248322 7406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72d4ea321b2f45bca9ed03e52012e9c4" candidate_uuid: "65d9be8a776a46f6a53142ae0accd7ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48c360f547f84f9881723197501c1cc0" is_pre_election: true
I20260927 02:34:04.248626 7543 raft_consensus.cc:2454] T 72d4ea321b2f45bca9ed03e52012e9c4 P 733db7c7c5b54c6495ba5a80d1e50104 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65d9be8a776a46f6a53142ae0accd7ee in term 0.
I20260927 02:34:04.248626 7406 raft_consensus.cc:2454] T 72d4ea321b2f45bca9ed03e52012e9c4 P 48c360f547f84f9881723197501c1cc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65d9be8a776a46f6a53142ae0accd7ee in term 0.
I20260927 02:34:04.249146 7613 leader_election.cc:302] T 72d4ea321b2f45bca9ed03e52012e9c4 P 65d9be8a776a46f6a53142ae0accd7ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 48c360f547f84f9881723197501c1cc0, 65d9be8a776a46f6a53142ae0accd7ee, 733db7c7c5b54c6495ba5a80d1e50104; no voters:
I20260927 02:34:04.249415 7767 raft_consensus.cc:2781] T 72d4ea321b2f45bca9ed03e52012e9c4 P 65d9be8a776a46f6a53142ae0accd7ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:34:04.249488 7767 raft_consensus.cc:482] T 72d4ea321b2f45bca9ed03e52012e9c4 P 65d9be8a776a46f6a53142ae0accd7ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:34:04.249536 7767 raft_consensus.cc:3037] T 72d4ea321b2f45bca9ed03e52012e9c4 P 65d9be8a776a46f6a53142ae0accd7ee [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:34:04.250533 7767 raft_consensus.cc:504] T 72d4ea321b2f45bca9ed03e52012e9c4 P 65d9be8a776a46f6a53142ae0accd7ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "733db7c7c5b54c6495ba5a80d1e50104" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37325 } } peers { permanent_uuid: "48c360f547f84f9881723197501c1cc0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46237 } } peers { permanent_uuid: "65d9be8a776a46f6a53142ae0accd7ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43921 } }
I20260927 02:34:04.250835 7767 leader_election.cc:288] T 72d4ea321b2f45bca9ed03e52012e9c4 P 65d9be8a776a46f6a53142ae0accd7ee [CANDIDATE]: Term 1 election: Requested vote from peers 733db7c7c5b54c6495ba5a80d1e50104 (127.0.98.2:37325), 48c360f547f84f9881723197501c1cc0 (127.0.98.1:46237)
I20260927 02:34:04.251142 7543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72d4ea321b2f45bca9ed03e52012e9c4" candidate_uuid: "65d9be8a776a46f6a53142ae0accd7ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "733db7c7c5b54c6495ba5a80d1e50104"
I20260927 02:34:04.251223 7406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72d4ea321b2f45bca9ed03e52012e9c4" candidate_uuid: "65d9be8a776a46f6a53142ae0accd7ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48c360f547f84f9881723197501c1cc0"
I20260927 02:34:04.251295 7543 raft_consensus.cc:3037] T 72d4ea321b2f45bca9ed03e52012e9c4 P 733db7c7c5b54c6495ba5a80d1e50104 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:34:04.251332 7406 raft_consensus.cc:3037] T 72d4ea321b2f45bca9ed03e52012e9c4 P 48c360f547f84f9881723197501c1cc0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:34:04.252179 7406 raft_consensus.cc:2454] T 72d4ea321b2f45bca9ed03e52012e9c4 P 48c360f547f84f9881723197501c1cc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65d9be8a776a46f6a53142ae0accd7ee in term 1.
I20260927 02:34:04.252323 7543 raft_consensus.cc:2454] T 72d4ea321b2f45bca9ed03e52012e9c4 P 733db7c7c5b54c6495ba5a80d1e50104 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65d9be8a776a46f6a53142ae0accd7ee in term 1.
I20260927 02:34:04.252470 7612 leader_election.cc:302] T 72d4ea321b2f45bca9ed03e52012e9c4 P 65d9be8a776a46f6a53142ae0accd7ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48c360f547f84f9881723197501c1cc0, 65d9be8a776a46f6a53142ae0accd7ee; no voters:
I20260927 02:34:04.252678 7767 raft_consensus.cc:2781] T 72d4ea321b2f45bca9ed03e52012e9c4 P 65d9be8a776a46f6a53142ae0accd7ee [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:34:04.252893 7767 raft_consensus.cc:686] T 72d4ea321b2f45bca9ed03e52012e9c4 P 65d9be8a776a46f6a53142ae0accd7ee [term 1 LEADER]: Becoming Leader. State: Replica: 65d9be8a776a46f6a53142ae0accd7ee, State: Running, Role: LEADER
I20260927 02:34:04.253232 7767 consensus_queue.cc:237] T 72d4ea321b2f45bca9ed03e52012e9c4 P 65d9be8a776a46f6a53142ae0accd7ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "733db7c7c5b54c6495ba5a80d1e50104" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37325 } } peers { permanent_uuid: "48c360f547f84f9881723197501c1cc0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46237 } } peers { permanent_uuid: "65d9be8a776a46f6a53142ae0accd7ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43921 } }
I20260927 02:34:04.256645 6488 catalog_manager.cc:5495] T 72d4ea321b2f45bca9ed03e52012e9c4 P 65d9be8a776a46f6a53142ae0accd7ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 65d9be8a776a46f6a53142ae0accd7ee (127.0.98.3). New cstate: current_term: 1 leader_uuid: "65d9be8a776a46f6a53142ae0accd7ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "733db7c7c5b54c6495ba5a80d1e50104" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48c360f547f84f9881723197501c1cc0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65d9be8a776a46f6a53142ae0accd7ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43921 } health_report { overall_health: HEALTHY } } }
I20260927 02:34:04.282670 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:04.279714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43012 (local address 127.0.98.3:43921)
0927 02:34:04.279823 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:04.279826 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:04.279838 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:34:04.279947 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:04.279949 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:04.280002 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:04.280108 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:34:04.280287 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:04.280770 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:04.281525 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:04.281728 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:04.281794 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:04.282006 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:34:04.282386 (+ 380us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:34:04.282475 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:04.282507 (+ 32us) server_negotiation.cc:299] Negotiation successful
0927 02:34:04.282554 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 02:34:04.287798 7543 raft_consensus.cc:1261] T 72d4ea321b2f45bca9ed03e52012e9c4 P 733db7c7c5b54c6495ba5a80d1e50104 [term 1 FOLLOWER]: Refusing update from remote peer 65d9be8a776a46f6a53142ae0accd7ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:34:04.287799 7406 raft_consensus.cc:1261] T 72d4ea321b2f45bca9ed03e52012e9c4 P 48c360f547f84f9881723197501c1cc0 [term 1 FOLLOWER]: Refusing update from remote peer 65d9be8a776a46f6a53142ae0accd7ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:34:04.288455 7768 consensus_queue.cc:1035] T 72d4ea321b2f45bca9ed03e52012e9c4 P 65d9be8a776a46f6a53142ae0accd7ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "733db7c7c5b54c6495ba5a80d1e50104" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:34:04.288614 7767 consensus_queue.cc:1035] T 72d4ea321b2f45bca9ed03e52012e9c4 P 65d9be8a776a46f6a53142ae0accd7ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "48c360f547f84f9881723197501c1cc0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:34:04.295890 7771 mvcc.cc:205] Tried to move back new op lower bound from 7333791515796623360 to 7333791515663679488. Current Snapshot: MvccSnapshot[applied={T|T < 7333791515796623360}]
I20260927 02:34:04.297089 7772 mvcc.cc:205] Tried to move back new op lower bound from 7333791515796623360 to 7333791515663679488. Current Snapshot: MvccSnapshot[applied={T|T < 7333791515796623360}]
I20260927 02:34:04.304258 6488 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:34:04.304414 6488 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:34:04.309005 6488 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aff7cb041f714927ae5a16a8dc279205: Sending DeleteTablet for 3 replicas of tablet 72d4ea321b2f45bca9ed03e52012e9c4
I20260927 02:34:04.309561 7523 tablet_service.cc:1514] Processing DeleteTablet for tablet 72d4ea321b2f45bca9ed03e52012e9c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:34:04 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42574
I20260927 02:34:04.309593 7386 tablet_service.cc:1514] Processing DeleteTablet for tablet 72d4ea321b2f45bca9ed03e52012e9c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:34:04 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36714
I20260927 02:34:04.309947 7660 tablet_service.cc:1514] Processing DeleteTablet for tablet 72d4ea321b2f45bca9ed03e52012e9c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:34:04 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43010
I20260927 02:34:04.310344 7783 tablet_replica.cc:330] T 72d4ea321b2f45bca9ed03e52012e9c4 P 65d9be8a776a46f6a53142ae0accd7ee: stopping tablet replica
I20260927 02:34:04.310614 7783 raft_consensus.cc:2229] T 72d4ea321b2f45bca9ed03e52012e9c4 P 65d9be8a776a46f6a53142ae0accd7ee [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:34:04.310972 7783 raft_consensus.cc:2258] T 72d4ea321b2f45bca9ed03e52012e9c4 P 65d9be8a776a46f6a53142ae0accd7ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:34:04.310842 7782 tablet_replica.cc:330] T 72d4ea321b2f45bca9ed03e52012e9c4 P 733db7c7c5b54c6495ba5a80d1e50104: stopping tablet replica
I20260927 02:34:04.311609 7782 raft_consensus.cc:2229] T 72d4ea321b2f45bca9ed03e52012e9c4 P 733db7c7c5b54c6495ba5a80d1e50104 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:34:04.312333 7782 raft_consensus.cc:2258] T 72d4ea321b2f45bca9ed03e52012e9c4 P 733db7c7c5b54c6495ba5a80d1e50104 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:34:04.312567 7783 ts_tablet_manager.cc:1905] T 72d4ea321b2f45bca9ed03e52012e9c4 P 65d9be8a776a46f6a53142ae0accd7ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:34:04.312649 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 6457
I20260927 02:34:04.315126 7783 ts_tablet_manager.cc:1918] T 72d4ea321b2f45bca9ed03e52012e9c4 P 65d9be8a776a46f6a53142ae0accd7ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:34:04.315223 7783 log.cc:1196] T 72d4ea321b2f45bca9ed03e52012e9c4 P 65d9be8a776a46f6a53142ae0accd7ee: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-2/wal/wals/72d4ea321b2f45bca9ed03e52012e9c4
I20260927 02:34:04.315496 7783 ts_tablet_manager.cc:1939] T 72d4ea321b2f45bca9ed03e52012e9c4 P 65d9be8a776a46f6a53142ae0accd7ee: Deleting consensus metadata
W20260927 02:34:04.319137 7614 connection.cc:535] server connection from 127.0.0.1:43010 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 02:34:04.319525 7781 tablet_replica.cc:330] T 72d4ea321b2f45bca9ed03e52012e9c4 P 48c360f547f84f9881723197501c1cc0: stopping tablet replica
I20260927 02:34:04.319828 7781 raft_consensus.cc:2229] T 72d4ea321b2f45bca9ed03e52012e9c4 P 48c360f547f84f9881723197501c1cc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:34:04.320122 7781 raft_consensus.cc:2258] T 72d4ea321b2f45bca9ed03e52012e9c4 P 48c360f547f84f9881723197501c1cc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:34:04.321563 7782 ts_tablet_manager.cc:1905] T 72d4ea321b2f45bca9ed03e52012e9c4 P 733db7c7c5b54c6495ba5a80d1e50104: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:34:04.322391 7781 ts_tablet_manager.cc:1905] T 72d4ea321b2f45bca9ed03e52012e9c4 P 48c360f547f84f9881723197501c1cc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:34:04.324110 7782 ts_tablet_manager.cc:1918] T 72d4ea321b2f45bca9ed03e52012e9c4 P 733db7c7c5b54c6495ba5a80d1e50104: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:34:04.324216 7782 log.cc:1196] T 72d4ea321b2f45bca9ed03e52012e9c4 P 733db7c7c5b54c6495ba5a80d1e50104: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-1/wal/wals/72d4ea321b2f45bca9ed03e52012e9c4
I20260927 02:34:04.324519 7782 ts_tablet_manager.cc:1939] T 72d4ea321b2f45bca9ed03e52012e9c4 P 733db7c7c5b54c6495ba5a80d1e50104: Deleting consensus metadata
W20260927 02:34:04.325613 7479 connection.cc:441] server connection from 127.0.0.1:42574 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42574 (request call id 1) could send its response
I20260927 02:34:04.328572 7781 ts_tablet_manager.cc:1918] T 72d4ea321b2f45bca9ed03e52012e9c4 P 48c360f547f84f9881723197501c1cc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:34:04.328887 7781 log.cc:1196] T 72d4ea321b2f45bca9ed03e52012e9c4 P 48c360f547f84f9881723197501c1cc0: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-0/wal/wals/72d4ea321b2f45bca9ed03e52012e9c4
I20260927 02:34:04.329243 7781 ts_tablet_manager.cc:1939] T 72d4ea321b2f45bca9ed03e52012e9c4 P 48c360f547f84f9881723197501c1cc0: Deleting consensus metadata
W20260927 02:34:04.330170 7340 connection.cc:441] server connection from 127.0.0.1:36714 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36714 (request call id 1) could send its response
I20260927 02:34:04.335995 7456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:04.322799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34422 (local address 127.0.98.60:36279)
0927 02:34:04.322905 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:04.322909 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:04.324285 (+ 1376us) server_negotiation.cc:407] Connection header received
0927 02:34:04.324460 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:04.324463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:04.324512 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:04.324565 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:04.325978 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:04.326655 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:04.327743 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:04.328079 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:04.328144 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:04.332584 (+ 4440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:04.332588 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:04.332591 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:04.332625 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:04.334270 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:04.334736 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:04.334739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:04.334741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:04.334818 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:04.335103 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:04.335107 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:04.335109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:04.335305 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:04.335400 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:04.335741 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 02:34:04.335845 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 02:34:04.336102 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:04.311070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60832 (local address 127.0.98.61:45743)
0927 02:34:04.311177 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:04.311181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:04.319733 (+ 8552us) server_negotiation.cc:407] Connection header received
0927 02:34:04.319916 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:04.319919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:04.319966 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:04.320013 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:04.322554 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:04.323369 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:04.326143 (+ 2774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:04.326480 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:04.326550 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:04.332603 (+ 6053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:04.332606 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:04.332609 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:04.332633 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:04.334460 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:04.334910 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:04.334914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:04.334915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:04.334992 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:04.335255 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:04.335258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:04.335260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:04.335447 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:04.335542 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:04.335849 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 02:34:04.335952 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":40}
I20260927 02:34:04.336555 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:04.323748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60848 (local address 127.0.98.61:45743)
0927 02:34:04.323853 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:04.323856 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:04.326370 (+ 2514us) server_negotiation.cc:407] Connection header received
0927 02:34:04.326406 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:04.326409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:04.326448 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:04.326509 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:04.327407 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:04.328115 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:04.329193 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:04.329501 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:04.329588 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:04.332584 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:04.332588 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:04.332591 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:04.332618 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:04.334952 (+ 2334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:04.335363 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:04.335366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:04.335368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:04.335441 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:04.335696 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:04.335700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:04.335701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:04.335877 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:04.335966 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:04.336311 (+ 345us) server_negotiation.cc:299] Negotiation successful
0927 02:34:04.336416 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":86,"server-negotiator.queue_time_us":41,"spinlock_wait_cycles":8704}
I20260927 02:34:04.344329 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:04.311363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34420 (local address 127.0.98.60:36279)
0927 02:34:04.311533 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:04.311537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:04.337249 (+ 25712us) server_negotiation.cc:407] Connection header received
0927 02:34:04.337389 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:04.337392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:04.337433 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:04.337475 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:04.338222 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:04.338683 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:04.339705 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:04.339978 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:04.340053 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:04.341445 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:04.341448 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:04.341450 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:04.341469 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:04.342850 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:04.343297 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:04.343300 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:04.343302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:04.343375 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:04.343599 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:04.343602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:04.343604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:04.343762 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:04.343860 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:04.344091 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 02:34:04.344193 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260927 02:34:05.302343 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:05.301860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.98.2:37269)
0927 02:34:05.302149 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:05.302242 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":118,"threads_started":1}
W20260927 02:34:05.302639 7590 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46055 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:46055: connect: Connection refused (error 111)
I20260927 02:34:05.303211 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:05.302861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.98.1:41571)
0927 02:34:05.303066 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:05.303135 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":115,"threads_started":1}
W20260927 02:34:05.303514 7453 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46055 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:46055: connect: Connection refused (error 111)
I20260927 02:34:05.304342 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:05.303958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.98.3:59217)
0927 02:34:05.304172 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:05.304247 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":57,"threads_started":1}
W20260927 02:34:05.304584 7727 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46055 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:46055: connect: Connection refused (error 111)
I20260927 02:34:05.808799 7795 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aff7cb041f714927ae5a16a8dc279205)
I20260927 02:34:05.809032 7795 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } peers { permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45743 } attrs { promote: false } } peers { permanent_uuid: "11b7f1d9d8944fd19085f2cc3c659137" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36279 } attrs { promote: false } }
I20260927 02:34:05.809376 7796 raft_consensus.cc:482] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aff7cb041f714927ae5a16a8dc279205)
I20260927 02:34:05.809537 7796 raft_consensus.cc:504] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } peers { permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45743 } attrs { promote: false } } peers { permanent_uuid: "11b7f1d9d8944fd19085f2cc3c659137" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36279 } attrs { promote: false } }
I20260927 02:34:05.810060 7795 leader_election.cc:288] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aff7cb041f714927ae5a16a8dc279205 (127.0.98.62:46055), 591735b7214747d5801020bb2bf64abd (127.0.98.61:45743)
I20260927 02:34:05.810359 7796 leader_election.cc:288] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aff7cb041f714927ae5a16a8dc279205 (127.0.98.62:46055), 11b7f1d9d8944fd19085f2cc3c659137 (127.0.98.60:36279)
I20260927 02:34:05.810742 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:05.810244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.0.1:58242)
0927 02:34:05.810583 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:05.810646 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":45,"threads_started":1}
I20260927 02:34:05.810874 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:05.810471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.0.1:58256)
0927 02:34:05.810719 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:05.810785 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":51,"threads_started":1}
W20260927 02:34:05.810910 7258 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46055: connect: Connection refused (error 111)
W20260927 02:34:05.811005 7074 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46055: connect: Connection refused (error 111)
I20260927 02:34:05.811801 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:05.811581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.0.1:58284)
0927 02:34:05.811698 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:05.811742 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 02:34:05.811810 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:05.811504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.0.1:58268)
0927 02:34:05.811692 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:05.811739 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
W20260927 02:34:05.811929 7074 leader_election.cc:334] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aff7cb041f714927ae5a16a8dc279205 (127.0.98.62:46055): Network error: Client connection negotiation failed: client connection to 127.0.98.62:46055: connect: Connection refused (error 111)
W20260927 02:34:05.811995 7258 leader_election.cc:334] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aff7cb041f714927ae5a16a8dc279205 (127.0.98.62:46055): Network error: Client connection negotiation failed: client connection to 127.0.98.62:46055: connect: Connection refused (error 111)
I20260927 02:34:05.814464 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:05.810909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34428 (local address 127.0.98.60:36279)
0927 02:34:05.811218 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:05.811222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:05.811513 (+ 291us) server_negotiation.cc:407] Connection header received
0927 02:34:05.811552 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:05.811554 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:05.811593 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:05.811661 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:34:05.812044 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:05.812591 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:05.813591 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:05.813977 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:05.814070 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:05.814219 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:05.814262 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 02:34:05.814324 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":254,"thread_start_us":61,"threads_started":1}
I20260927 02:34:05.814466 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:05.810585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36279 (local address 127.0.0.1:34428)
0927 02:34:05.811325 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:05.811337 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:05.811437 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:05.811759 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:05.811762 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:05.811777 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:34:05.811865 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:05.811871 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:05.812685 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:05.812688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:05.813489 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:05.813497 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:05.814079 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:05.814083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:05.814167 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:05.814188 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 02:34:05.814242 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 02:34:05.814306 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"spinlock_wait_cycles":4992,"thread_start_us":60,"threads_started":1}
I20260927 02:34:05.814483 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:05.810725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45743 (local address 127.0.0.1:60852)
0927 02:34:05.810915 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:05.810927 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:05.811042 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:05.811577 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:05.811580 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:05.811600 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:34:05.811725 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:05.811732 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:05.812593 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:05.812596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:05.813417 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:05.813426 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:05.814015 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:05.814019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:05.814120 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:05.814203 (+ 83us) client_negotiation.cc:769] Sending connection context
0927 02:34:05.814266 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 02:34:05.814336 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":44,"thread_start_us":53,"threads_started":1}
I20260927 02:34:05.814483 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:05.810548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60852 (local address 127.0.98.61:45743)
0927 02:34:05.811300 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:05.811304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:05.811319 (+ 15us) server_negotiation.cc:407] Connection header received
0927 02:34:05.811386 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:05.811388 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:05.811447 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:05.811825 (+ 378us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:34:05.811909 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:05.812496 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:05.813521 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:05.813918 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:05.814005 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:05.814062 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:05.814290 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 02:34:05.814355 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"spinlock_wait_cycles":4608,"thread_start_us":54,"threads_started":1}
I20260927 02:34:05.814870 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "591735b7214747d5801020bb2bf64abd" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "11b7f1d9d8944fd19085f2cc3c659137" is_pre_election: true
I20260927 02:34:05.815066 7282 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 591735b7214747d5801020bb2bf64abd in term 1.
I20260927 02:34:05.815112 7098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "11b7f1d9d8944fd19085f2cc3c659137" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "591735b7214747d5801020bb2bf64abd" is_pre_election: true
I20260927 02:34:05.815275 7098 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11b7f1d9d8944fd19085f2cc3c659137 in term 1.
I20260927 02:34:05.815371 7076 leader_election.cc:302] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 11b7f1d9d8944fd19085f2cc3c659137, 591735b7214747d5801020bb2bf64abd; no voters: aff7cb041f714927ae5a16a8dc279205
I20260927 02:34:05.815555 7796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 02:34:05.815578 7258 leader_election.cc:302] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 11b7f1d9d8944fd19085f2cc3c659137, 591735b7214747d5801020bb2bf64abd; no voters: aff7cb041f714927ae5a16a8dc279205
I20260927 02:34:05.815608 7796 raft_consensus.cc:482] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [term 1 FOLLOWER]: Starting leader election (detected failure of leader aff7cb041f714927ae5a16a8dc279205)
I20260927 02:34:05.815646 7796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [term 1 FOLLOWER]: Advancing to term 2
I20260927 02:34:05.815750 7795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 02:34:05.815820 7795 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137 [term 1 FOLLOWER]: Starting leader election (detected failure of leader aff7cb041f714927ae5a16a8dc279205)
I20260927 02:34:05.815869 7795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137 [term 1 FOLLOWER]: Advancing to term 2
I20260927 02:34:05.816833 7796 raft_consensus.cc:504] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } peers { permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45743 } attrs { promote: false } } peers { permanent_uuid: "11b7f1d9d8944fd19085f2cc3c659137" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36279 } attrs { promote: false } }
I20260927 02:34:05.816875 7795 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } peers { permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45743 } attrs { promote: false } } peers { permanent_uuid: "11b7f1d9d8944fd19085f2cc3c659137" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36279 } attrs { promote: false } }
I20260927 02:34:05.817195 7795 leader_election.cc:288] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137 [CANDIDATE]: Term 2 election: Requested vote from peers aff7cb041f714927ae5a16a8dc279205 (127.0.98.62:46055), 591735b7214747d5801020bb2bf64abd (127.0.98.61:45743)
I20260927 02:34:05.817207 7796 leader_election.cc:288] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [CANDIDATE]: Term 2 election: Requested vote from peers aff7cb041f714927ae5a16a8dc279205 (127.0.98.62:46055), 11b7f1d9d8944fd19085f2cc3c659137 (127.0.98.60:36279)
I20260927 02:34:05.817504 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:05.817292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.0.1:58290)
0927 02:34:05.817371 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:05.817445 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260927 02:34:05.817559 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "591735b7214747d5801020bb2bf64abd" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "11b7f1d9d8944fd19085f2cc3c659137"
I20260927 02:34:05.817687 7282 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 591735b7214747d5801020bb2bf64abd in current term 2: Already voted for candidate 11b7f1d9d8944fd19085f2cc3c659137 in this term.
I20260927 02:34:05.817759 7098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "11b7f1d9d8944fd19085f2cc3c659137" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "591735b7214747d5801020bb2bf64abd"
I20260927 02:34:05.817904 7098 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 11b7f1d9d8944fd19085f2cc3c659137 in current term 2: Already voted for candidate 591735b7214747d5801020bb2bf64abd in this term.
I20260927 02:34:05.817958 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:05.817481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.0.1:58304)
0927 02:34:05.817856 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:05.817903 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319}
I20260927 02:34:05.818439 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:05.818299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.0.1:58306)
0927 02:34:05.818359 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:05.818396 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260927 02:34:05.818528 7258 leader_election.cc:334] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer aff7cb041f714927ae5a16a8dc279205 (127.0.98.62:46055): Network error: Client connection negotiation failed: client connection to 127.0.98.62:46055: connect: Connection refused (error 111)
I20260927 02:34:05.818598 7258 leader_election.cc:302] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 11b7f1d9d8944fd19085f2cc3c659137; no voters: 591735b7214747d5801020bb2bf64abd, aff7cb041f714927ae5a16a8dc279205
I20260927 02:34:05.818711 7795 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 02:34:05.818801 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:05.818623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.0.1:58316)
0927 02:34:05.818704 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:05.818751 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260927 02:34:05.818879 7074 leader_election.cc:334] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer aff7cb041f714927ae5a16a8dc279205 (127.0.98.62:46055): Network error: Client connection negotiation failed: client connection to 127.0.98.62:46055: connect: Connection refused (error 111)
I20260927 02:34:05.818946 7074 leader_election.cc:302] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 591735b7214747d5801020bb2bf64abd; no voters: 11b7f1d9d8944fd19085f2cc3c659137, aff7cb041f714927ae5a16a8dc279205
I20260927 02:34:05.819061 7796 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 02:34:06.303565 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:06.303138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.98.2:34001)
0927 02:34:06.303401 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:06.303473 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":88,"threads_started":1}
I20260927 02:34:06.304315 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:06.303988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.98.1:56979)
0927 02:34:06.304181 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:06.304235 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":66,"threads_started":1}
I20260927 02:34:06.305428 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:06.305049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.98.3:54581)
0927 02:34:06.305288 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:06.305339 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":55,"threads_started":1}
I20260927 02:34:07.304723 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:07.304337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.98.2:59991)
0927 02:34:07.304557 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:07.304622 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":67,"threads_started":1}
I20260927 02:34:07.305222 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:07.304822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.98.1:36855)
0927 02:34:07.305080 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:07.305137 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":56,"threads_started":1}
I20260927 02:34:07.306289 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:07.305937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.98.3:39465)
0927 02:34:07.306157 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:07.306220 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":63,"threads_started":1}
I20260927 02:34:07.406365 7811 raft_consensus.cc:482] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:34:07.406520 7811 raft_consensus.cc:504] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } peers { permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45743 } attrs { promote: false } } peers { permanent_uuid: "11b7f1d9d8944fd19085f2cc3c659137" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36279 } attrs { promote: false } }
I20260927 02:34:07.406906 7811 leader_election.cc:288] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers aff7cb041f714927ae5a16a8dc279205 (127.0.98.62:46055), 11b7f1d9d8944fd19085f2cc3c659137 (127.0.98.60:36279)
I20260927 02:34:07.407306 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "591735b7214747d5801020bb2bf64abd" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "11b7f1d9d8944fd19085f2cc3c659137" is_pre_election: true
I20260927 02:34:07.407430 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:07.407085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.0.1:58544)
0927 02:34:07.407288 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:07.407341 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":65,"threads_started":1}
I20260927 02:34:07.407434 7282 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 591735b7214747d5801020bb2bf64abd in term 2.
I20260927 02:34:07.407717 7076 leader_election.cc:302] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11b7f1d9d8944fd19085f2cc3c659137, 591735b7214747d5801020bb2bf64abd; no voters:
I20260927 02:34:07.407881 7811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 02:34:07.407943 7811 raft_consensus.cc:482] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:34:07.407970 7811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [term 2 FOLLOWER]: Advancing to term 3
I20260927 02:34:07.408981 7811 raft_consensus.cc:504] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } peers { permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45743 } attrs { promote: false } } peers { permanent_uuid: "11b7f1d9d8944fd19085f2cc3c659137" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36279 } attrs { promote: false } }
I20260927 02:34:07.409356 7811 leader_election.cc:288] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [CANDIDATE]: Term 3 election: Requested vote from peers aff7cb041f714927ae5a16a8dc279205 (127.0.98.62:46055), 11b7f1d9d8944fd19085f2cc3c659137 (127.0.98.60:36279)
I20260927 02:34:07.409361 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:07.409130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.0.1:58556)
0927 02:34:07.409237 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:07.409291 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260927 02:34:07.409578 7074 leader_election.cc:334] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aff7cb041f714927ae5a16a8dc279205 (127.0.98.62:46055): Network error: Client connection negotiation failed: client connection to 127.0.98.62:46055: connect: Connection refused (error 111)
I20260927 02:34:07.409693 7282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "591735b7214747d5801020bb2bf64abd" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "11b7f1d9d8944fd19085f2cc3c659137"
I20260927 02:34:07.409811 7282 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137 [term 2 FOLLOWER]: Advancing to term 3
I20260927 02:34:07.410041 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:07.409864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.0.1:58558)
0927 02:34:07.409944 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:07.409988 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260927 02:34:07.410198 7074 leader_election.cc:334] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer aff7cb041f714927ae5a16a8dc279205 (127.0.98.62:46055): Network error: Client connection negotiation failed: client connection to 127.0.98.62:46055: connect: Connection refused (error 111)
I20260927 02:34:07.410744 7282 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 591735b7214747d5801020bb2bf64abd in term 3.
I20260927 02:34:07.411006 7076 leader_election.cc:302] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 11b7f1d9d8944fd19085f2cc3c659137, 591735b7214747d5801020bb2bf64abd; no voters: aff7cb041f714927ae5a16a8dc279205
I20260927 02:34:07.411162 7811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [term 3 FOLLOWER]: Leader election won for term 3
I20260927 02:34:07.411341 7811 raft_consensus.cc:686] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [term 3 LEADER]: Becoming Leader. State: Replica: 591735b7214747d5801020bb2bf64abd, State: Running, Role: LEADER
I20260927 02:34:07.411597 7811 consensus_queue.cc:237] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } peers { permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45743 } attrs { promote: false } } peers { permanent_uuid: "11b7f1d9d8944fd19085f2cc3c659137" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36279 } attrs { promote: false } }
I20260927 02:34:07.412745 7814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 591735b7214747d5801020bb2bf64abd. Latest consensus state: current_term: 3 leader_uuid: "591735b7214747d5801020bb2bf64abd" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } peers { permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45743 } attrs { promote: false } } peers { permanent_uuid: "11b7f1d9d8944fd19085f2cc3c659137" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36279 } attrs { promote: false } } }
I20260927 02:34:07.412837 7814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [sys.catalog]: This master's current role is: LEADER
I20260927 02:34:07.413077 7816 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:34:07.415103 7816 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:34:07.415475 7816 catalog_manager.cc:1259] Loaded cluster ID: b32f0fff8a2b4f56877e69a9f46fb0b9
I20260927 02:34:07.415518 7816 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:34:07.416045 7816 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:34:07.416405 7816 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd: Loaded TSK: 0
I20260927 02:34:07.417047 7816 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:34:07.448041 7089 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60848:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:34:07.450874 7089 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:34:07.453477 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:07.453248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.0.1:58582)
0927 02:34:07.453350 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:07.453406 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 02:34:07.453539 7282 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137 [term 3 FOLLOWER]: Refusing update from remote peer 591735b7214747d5801020bb2bf64abd: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260927 02:34:07.453994 7814 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [LEADER]: Connected to new peer: Peer: permanent_uuid: "11b7f1d9d8944fd19085f2cc3c659137" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260927 02:34:07.454109 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:07.453900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.0.1:58590)
0927 02:34:07.454013 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:07.454059 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 02:34:07.454210 7074 consensus_peers.cc:487] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd -> Peer aff7cb041f714927ae5a16a8dc279205 (127.0.98.62:46055): Couldn't send request to peer aff7cb041f714927ae5a16a8dc279205. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46055: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 02:34:07.455313 7818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 591735b7214747d5801020bb2bf64abd. Latest consensus state: current_term: 3 leader_uuid: "591735b7214747d5801020bb2bf64abd" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } peers { permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45743 } attrs { promote: false } } peers { permanent_uuid: "11b7f1d9d8944fd19085f2cc3c659137" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36279 } attrs { promote: false } } }
I20260927 02:34:07.455418 7818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:34:07.456348 7814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "591735b7214747d5801020bb2bf64abd" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } peers { permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45743 } attrs { promote: false } } peers { permanent_uuid: "11b7f1d9d8944fd19085f2cc3c659137" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36279 } attrs { promote: false } } }
I20260927 02:34:07.456435 7814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd [sys.catalog]: This master's current role is: LEADER
I20260927 02:34:07.456651 7818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "591735b7214747d5801020bb2bf64abd" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aff7cb041f714927ae5a16a8dc279205" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46055 } } peers { permanent_uuid: "591735b7214747d5801020bb2bf64abd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45743 } attrs { promote: false } } peers { permanent_uuid: "11b7f1d9d8944fd19085f2cc3c659137" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36279 } attrs { promote: false } } }
I20260927 02:34:07.456759 7818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11b7f1d9d8944fd19085f2cc3c659137 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:34:07.466099 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:07.461621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43921 (local address 127.0.0.1:43020)
0927 02:34:07.462016 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:07.462029 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:07.462137 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:07.462364 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:07.462367 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:07.462384 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:34:07.462476 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:07.462482 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:07.463540 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:07.463544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:07.464751 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:07.464760 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:07.465741 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:07.465746 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:07.465849 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:07.465877 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 02:34:07.465927 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 02:34:07.465994 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347}
I20260927 02:34:07.466547 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:07.461731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43020 (local address 127.0.98.3:43921)
0927 02:34:07.462016 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:07.462021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:07.462099 (+ 78us) server_negotiation.cc:407] Connection header received
0927 02:34:07.462216 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:07.462219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:07.462279 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:07.462369 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:34:07.462595 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:07.463425 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:07.464890 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:07.465600 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:07.466271 (+ 671us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:07.466332 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:07.466363 (+ 31us) server_negotiation.cc:299] Negotiation successful
0927 02:34:07.466427 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":58,"threads_started":1}
I20260927 02:34:07.466791 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:07.461847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37325 (local address 127.0.0.1:42586)
0927 02:34:07.462073 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:07.462086 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:07.462181 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:07.463186 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:07.463189 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:07.463202 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:34:07.463271 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:07.463277 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:07.464268 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:07.464272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:07.465526 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:07.465536 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:07.466448 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:07.466451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:07.466530 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:07.466552 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 02:34:07.466611 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 02:34:07.466673 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":22,"thread_start_us":66,"threads_started":1}
I20260927 02:34:07.467011 7660 tablet_service.cc:1467] Processing CreateTablet for tablet fc76faad05a14bceb55e1848e6e01b35 (DEFAULT_TABLE table=test-table [id=d1110b75e20f44caa7f6ce07af18890e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:34:07.466910 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:07.462347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46237 (local address 127.0.0.1:36718)
0927 02:34:07.462542 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:07.462565 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:07.462640 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:07.463092 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:07.463094 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:07.463108 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:34:07.463187 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:07.463192 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:07.464269 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:07.464272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:07.465689 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:07.465696 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:07.466617 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:07.466620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:07.466695 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:07.466714 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 02:34:07.466754 (+ 40us) client_negotiation.cc:240] Negotiation successful
0927 02:34:07.466806 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":61,"thread_start_us":79,"threads_started":1}
I20260927 02:34:07.467278 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:07.461961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42586 (local address 127.0.98.2:37325)
0927 02:34:07.462902 (+ 941us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:07.462906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:07.462919 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:34:07.462961 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:07.462964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:07.463006 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:07.463095 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:34:07.463376 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:07.464151 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:07.465657 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:07.466249 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:07.466338 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:07.466395 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:07.466941 (+ 546us) server_negotiation.cc:299] Negotiation successful
0927 02:34:07.467015 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":875,"thread_start_us":57,"threads_started":1}
I20260927 02:34:07.467296 7660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc76faad05a14bceb55e1848e6e01b35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:34:07.467540 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:07.462659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36718 (local address 127.0.98.1:46237)
0927 02:34:07.462869 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:07.462873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:07.462885 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:34:07.462932 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:07.462934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:07.462977 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:07.463324 (+ 347us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:34:07.463423 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:07.464156 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:07.465852 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:07.466489 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:07.467001 (+ 512us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:07.467059 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:07.467093 (+ 34us) server_negotiation.cc:299] Negotiation successful
0927 02:34:07.467424 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":77,"threads_started":1}
I20260927 02:34:07.467681 7523 tablet_service.cc:1467] Processing CreateTablet for tablet fc76faad05a14bceb55e1848e6e01b35 (DEFAULT_TABLE table=test-table [id=d1110b75e20f44caa7f6ce07af18890e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:34:07.467916 7523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc76faad05a14bceb55e1848e6e01b35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:34:07.469645 7829 tablet_bootstrap.cc:492] T fc76faad05a14bceb55e1848e6e01b35 P 65d9be8a776a46f6a53142ae0accd7ee: Bootstrap starting.
I20260927 02:34:07.470026 7386 tablet_service.cc:1467] Processing CreateTablet for tablet fc76faad05a14bceb55e1848e6e01b35 (DEFAULT_TABLE table=test-table [id=d1110b75e20f44caa7f6ce07af18890e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:34:07.470244 7386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc76faad05a14bceb55e1848e6e01b35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:34:07.470394 7829 tablet_bootstrap.cc:654] T fc76faad05a14bceb55e1848e6e01b35 P 65d9be8a776a46f6a53142ae0accd7ee: Neither blocks nor log segments found. Creating new log.
I20260927 02:34:07.470922 7830 tablet_bootstrap.cc:492] T fc76faad05a14bceb55e1848e6e01b35 P 733db7c7c5b54c6495ba5a80d1e50104: Bootstrap starting.
I20260927 02:34:07.471443 7829 tablet_bootstrap.cc:492] T fc76faad05a14bceb55e1848e6e01b35 P 65d9be8a776a46f6a53142ae0accd7ee: No bootstrap required, opened a new log
I20260927 02:34:07.471541 7829 ts_tablet_manager.cc:1397] T fc76faad05a14bceb55e1848e6e01b35 P 65d9be8a776a46f6a53142ae0accd7ee: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:34:07.471748 7830 tablet_bootstrap.cc:654] T fc76faad05a14bceb55e1848e6e01b35 P 733db7c7c5b54c6495ba5a80d1e50104: Neither blocks nor log segments found. Creating new log.
I20260927 02:34:07.472007 7829 raft_consensus.cc:348] T fc76faad05a14bceb55e1848e6e01b35 P 65d9be8a776a46f6a53142ae0accd7ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d9be8a776a46f6a53142ae0accd7ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43921 } } peers { permanent_uuid: "733db7c7c5b54c6495ba5a80d1e50104" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37325 } } peers { permanent_uuid: "48c360f547f84f9881723197501c1cc0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46237 } }
I20260927 02:34:07.472147 7829 raft_consensus.cc:374] T fc76faad05a14bceb55e1848e6e01b35 P 65d9be8a776a46f6a53142ae0accd7ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:34:07.472203 7829 raft_consensus.cc:729] T fc76faad05a14bceb55e1848e6e01b35 P 65d9be8a776a46f6a53142ae0accd7ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65d9be8a776a46f6a53142ae0accd7ee, State: Initialized, Role: FOLLOWER
I20260927 02:34:07.472203 7832 tablet_bootstrap.cc:492] T fc76faad05a14bceb55e1848e6e01b35 P 48c360f547f84f9881723197501c1cc0: Bootstrap starting.
I20260927 02:34:07.472352 7829 consensus_queue.cc:260] T fc76faad05a14bceb55e1848e6e01b35 P 65d9be8a776a46f6a53142ae0accd7ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d9be8a776a46f6a53142ae0accd7ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43921 } } peers { permanent_uuid: "733db7c7c5b54c6495ba5a80d1e50104" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37325 } } peers { permanent_uuid: "48c360f547f84f9881723197501c1cc0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46237 } }
I20260927 02:34:07.472990 7829 ts_tablet_manager.cc:1428] T fc76faad05a14bceb55e1848e6e01b35 P 65d9be8a776a46f6a53142ae0accd7ee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:34:07.473006 7832 tablet_bootstrap.cc:654] T fc76faad05a14bceb55e1848e6e01b35 P 48c360f547f84f9881723197501c1cc0: Neither blocks nor log segments found. Creating new log.
I20260927 02:34:07.473486 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:07.473244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.98.3:46651)
0927 02:34:07.473379 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:07.473428 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 02:34:07.473963 7830 tablet_bootstrap.cc:492] T fc76faad05a14bceb55e1848e6e01b35 P 733db7c7c5b54c6495ba5a80d1e50104: No bootstrap required, opened a new log
I20260927 02:34:07.474046 7830 ts_tablet_manager.cc:1397] T fc76faad05a14bceb55e1848e6e01b35 P 733db7c7c5b54c6495ba5a80d1e50104: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 02:34:07.474175 7832 tablet_bootstrap.cc:492] T fc76faad05a14bceb55e1848e6e01b35 P 48c360f547f84f9881723197501c1cc0: No bootstrap required, opened a new log
I20260927 02:34:07.474268 7832 ts_tablet_manager.cc:1397] T fc76faad05a14bceb55e1848e6e01b35 P 48c360f547f84f9881723197501c1cc0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260927 02:34:07.474476 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:07.474258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.98.3:59581)
0927 02:34:07.474341 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:07.474386 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 02:34:07.474448 7830 raft_consensus.cc:348] T fc76faad05a14bceb55e1848e6e01b35 P 733db7c7c5b54c6495ba5a80d1e50104 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d9be8a776a46f6a53142ae0accd7ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43921 } } peers { permanent_uuid: "733db7c7c5b54c6495ba5a80d1e50104" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37325 } } peers { permanent_uuid: "48c360f547f84f9881723197501c1cc0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46237 } }
I20260927 02:34:07.474542 7830 raft_consensus.cc:374] T fc76faad05a14bceb55e1848e6e01b35 P 733db7c7c5b54c6495ba5a80d1e50104 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:34:07.474584 7830 raft_consensus.cc:729] T fc76faad05a14bceb55e1848e6e01b35 P 733db7c7c5b54c6495ba5a80d1e50104 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 733db7c7c5b54c6495ba5a80d1e50104, State: Initialized, Role: FOLLOWER
I20260927 02:34:07.474725 7830 consensus_queue.cc:260] T fc76faad05a14bceb55e1848e6e01b35 P 733db7c7c5b54c6495ba5a80d1e50104 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d9be8a776a46f6a53142ae0accd7ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43921 } } peers { permanent_uuid: "733db7c7c5b54c6495ba5a80d1e50104" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37325 } } peers { permanent_uuid: "48c360f547f84f9881723197501c1cc0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46237 } }
I20260927 02:34:07.475069 7830 ts_tablet_manager.cc:1428] T fc76faad05a14bceb55e1848e6e01b35 P 733db7c7c5b54c6495ba5a80d1e50104: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:34:07.475014 7832 raft_consensus.cc:348] T fc76faad05a14bceb55e1848e6e01b35 P 48c360f547f84f9881723197501c1cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d9be8a776a46f6a53142ae0accd7ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43921 } } peers { permanent_uuid: "733db7c7c5b54c6495ba5a80d1e50104" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37325 } } peers { permanent_uuid: "48c360f547f84f9881723197501c1cc0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46237 } }
I20260927 02:34:07.475188 7832 raft_consensus.cc:374] T fc76faad05a14bceb55e1848e6e01b35 P 48c360f547f84f9881723197501c1cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:34:07.475267 7832 raft_consensus.cc:729] T fc76faad05a14bceb55e1848e6e01b35 P 48c360f547f84f9881723197501c1cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48c360f547f84f9881723197501c1cc0, State: Initialized, Role: FOLLOWER
I20260927 02:34:07.475910 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:07.475675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.98.2:47775)
0927 02:34:07.475785 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:07.475841 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 02:34:07.475620 7832 consensus_queue.cc:260] T fc76faad05a14bceb55e1848e6e01b35 P 48c360f547f84f9881723197501c1cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d9be8a776a46f6a53142ae0accd7ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43921 } } peers { permanent_uuid: "733db7c7c5b54c6495ba5a80d1e50104" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37325 } } peers { permanent_uuid: "48c360f547f84f9881723197501c1cc0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46237 } }
I20260927 02:34:07.476441 7832 ts_tablet_manager.cc:1428] T fc76faad05a14bceb55e1848e6e01b35 P 48c360f547f84f9881723197501c1cc0: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260927 02:34:07.477909 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:07.477661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.98.1:51027)
0927 02:34:07.477799 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:07.477845 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 02:34:07.478479 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:07.478274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.98.2:53531)
0927 02:34:07.478372 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:07.478420 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 02:34:07.479611 7728 heartbeater.cc:503] Master 127.0.98.61:45743 was elected leader, sending a full tablet report...
I20260927 02:34:07.481142 7591 heartbeater.cc:503] Master 127.0.98.61:45743 was elected leader, sending a full tablet report...
I20260927 02:34:07.485337 7454 heartbeater.cc:503] Master 127.0.98.61:45743 was elected leader, sending a full tablet report...
I20260927 02:34:07.485621 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:07.485413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.98.1:35983)
0927 02:34:07.485513 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:07.485560 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 02:34:07.545996 7837 raft_consensus.cc:482] T fc76faad05a14bceb55e1848e6e01b35 P 48c360f547f84f9881723197501c1cc0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:34:07.546140 7837 raft_consensus.cc:504] T fc76faad05a14bceb55e1848e6e01b35 P 48c360f547f84f9881723197501c1cc0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d9be8a776a46f6a53142ae0accd7ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43921 } } peers { permanent_uuid: "733db7c7c5b54c6495ba5a80d1e50104" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37325 } } peers { permanent_uuid: "48c360f547f84f9881723197501c1cc0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46237 } }
I20260927 02:34:07.547159 7837 leader_election.cc:288] T fc76faad05a14bceb55e1848e6e01b35 P 48c360f547f84f9881723197501c1cc0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65d9be8a776a46f6a53142ae0accd7ee (127.0.98.3:43921), 733db7c7c5b54c6495ba5a80d1e50104 (127.0.98.2:37325)
I20260927 02:34:07.550659 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:07.547382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43921 (local address 127.0.98.1:60413)
0927 02:34:07.547497 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:07.547511 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:07.547616 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:07.547879 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:07.547882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:07.547899 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:34:07.548008 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:07.548013 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:07.548715 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:07.548718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:07.549648 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:07.549657 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:07.550262 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:07.550266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:07.550362 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:07.550381 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 02:34:07.550446 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 02:34:07.550510 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":2}
I20260927 02:34:07.550652 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:07.547470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60413 (local address 127.0.98.3:43921)
0927 02:34:07.547579 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:07.547583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:07.547596 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:34:07.547741 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:07.547744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:07.547787 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:07.547856 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:34:07.548106 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:07.548615 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:07.549763 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:07.550142 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:07.550232 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:07.550293 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:07.550472 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 02:34:07.550537 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 02:34:07.550705 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:07.547475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50723 (local address 127.0.98.2:37325)
0927 02:34:07.547594 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:07.547598 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:07.547841 (+ 243us) server_negotiation.cc:407] Connection header received
0927 02:34:07.547930 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:07.547933 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:07.547976 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:07.548035 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:34:07.548274 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:07.548773 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:07.549843 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:07.550222 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:07.550309 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:07.550366 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:07.550525 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 02:34:07.550597 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 02:34:07.550720 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:07.547390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37325 (local address 127.0.98.1:50723)
0927 02:34:07.547756 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:07.547766 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:07.547846 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:07.548061 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:07.548064 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:07.548078 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:34:07.548174 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:07.548179 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:07.548870 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:07.548874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:07.549722 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:07.549731 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:07.550321 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:07.550324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:07.550416 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:07.550437 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 02:34:07.550500 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 02:34:07.550566 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":110,"thread_start_us":100,"threads_started":1}
I20260927 02:34:07.551165 7543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc76faad05a14bceb55e1848e6e01b35" candidate_uuid: "48c360f547f84f9881723197501c1cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "733db7c7c5b54c6495ba5a80d1e50104" is_pre_election: true
I20260927 02:34:07.551214 7680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc76faad05a14bceb55e1848e6e01b35" candidate_uuid: "48c360f547f84f9881723197501c1cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65d9be8a776a46f6a53142ae0accd7ee" is_pre_election: true
I20260927 02:34:07.551286 7543 raft_consensus.cc:2454] T fc76faad05a14bceb55e1848e6e01b35 P 733db7c7c5b54c6495ba5a80d1e50104 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48c360f547f84f9881723197501c1cc0 in term 0.
I20260927 02:34:07.551419 7680 raft_consensus.cc:2454] T fc76faad05a14bceb55e1848e6e01b35 P 65d9be8a776a46f6a53142ae0accd7ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48c360f547f84f9881723197501c1cc0 in term 0.
I20260927 02:34:07.551658 7340 leader_election.cc:302] T fc76faad05a14bceb55e1848e6e01b35 P 48c360f547f84f9881723197501c1cc0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48c360f547f84f9881723197501c1cc0, 733db7c7c5b54c6495ba5a80d1e50104; no voters:
I20260927 02:34:07.551887 7837 raft_consensus.cc:2781] T fc76faad05a14bceb55e1848e6e01b35 P 48c360f547f84f9881723197501c1cc0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:34:07.551935 7837 raft_consensus.cc:482] T fc76faad05a14bceb55e1848e6e01b35 P 48c360f547f84f9881723197501c1cc0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:34:07.551961 7837 raft_consensus.cc:3037] T fc76faad05a14bceb55e1848e6e01b35 P 48c360f547f84f9881723197501c1cc0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:34:07.552965 7837 raft_consensus.cc:504] T fc76faad05a14bceb55e1848e6e01b35 P 48c360f547f84f9881723197501c1cc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d9be8a776a46f6a53142ae0accd7ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43921 } } peers { permanent_uuid: "733db7c7c5b54c6495ba5a80d1e50104" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37325 } } peers { permanent_uuid: "48c360f547f84f9881723197501c1cc0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46237 } }
I20260927 02:34:07.553282 7837 leader_election.cc:288] T fc76faad05a14bceb55e1848e6e01b35 P 48c360f547f84f9881723197501c1cc0 [CANDIDATE]: Term 1 election: Requested vote from peers 65d9be8a776a46f6a53142ae0accd7ee (127.0.98.3:43921), 733db7c7c5b54c6495ba5a80d1e50104 (127.0.98.2:37325)
I20260927 02:34:07.553591 7543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc76faad05a14bceb55e1848e6e01b35" candidate_uuid: "48c360f547f84f9881723197501c1cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "733db7c7c5b54c6495ba5a80d1e50104"
I20260927 02:34:07.553714 7543 raft_consensus.cc:3037] T fc76faad05a14bceb55e1848e6e01b35 P 733db7c7c5b54c6495ba5a80d1e50104 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:34:07.554688 7543 raft_consensus.cc:2454] T fc76faad05a14bceb55e1848e6e01b35 P 733db7c7c5b54c6495ba5a80d1e50104 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48c360f547f84f9881723197501c1cc0 in term 1.
I20260927 02:34:07.554898 7680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc76faad05a14bceb55e1848e6e01b35" candidate_uuid: "48c360f547f84f9881723197501c1cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65d9be8a776a46f6a53142ae0accd7ee"
I20260927 02:34:07.554939 7340 leader_election.cc:302] T fc76faad05a14bceb55e1848e6e01b35 P 48c360f547f84f9881723197501c1cc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48c360f547f84f9881723197501c1cc0, 733db7c7c5b54c6495ba5a80d1e50104; no voters:
I20260927 02:34:07.554999 7680 raft_consensus.cc:3037] T fc76faad05a14bceb55e1848e6e01b35 P 65d9be8a776a46f6a53142ae0accd7ee [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:34:07.555085 7837 raft_consensus.cc:2781] T fc76faad05a14bceb55e1848e6e01b35 P 48c360f547f84f9881723197501c1cc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:34:07.555245 7837 raft_consensus.cc:686] T fc76faad05a14bceb55e1848e6e01b35 P 48c360f547f84f9881723197501c1cc0 [term 1 LEADER]: Becoming Leader. State: Replica: 48c360f547f84f9881723197501c1cc0, State: Running, Role: LEADER
I20260927 02:34:07.555588 7837 consensus_queue.cc:237] T fc76faad05a14bceb55e1848e6e01b35 P 48c360f547f84f9881723197501c1cc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d9be8a776a46f6a53142ae0accd7ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43921 } } peers { permanent_uuid: "733db7c7c5b54c6495ba5a80d1e50104" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37325 } } peers { permanent_uuid: "48c360f547f84f9881723197501c1cc0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46237 } }
I20260927 02:34:07.556030 7680 raft_consensus.cc:2454] T fc76faad05a14bceb55e1848e6e01b35 P 65d9be8a776a46f6a53142ae0accd7ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48c360f547f84f9881723197501c1cc0 in term 1.
I20260927 02:34:07.557561 7081 catalog_manager.cc:5495] T fc76faad05a14bceb55e1848e6e01b35 P 48c360f547f84f9881723197501c1cc0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48c360f547f84f9881723197501c1cc0 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "48c360f547f84f9881723197501c1cc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d9be8a776a46f6a53142ae0accd7ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "733db7c7c5b54c6495ba5a80d1e50104" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48c360f547f84f9881723197501c1cc0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46237 } health_report { overall_health: HEALTHY } } }
I20260927 02:34:07.560748 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:07.559477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.98.1:60317)
0927 02:34:07.560630 (+ 1153us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:07.560684 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1090}
I20260927 02:34:07.569984 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:07.566892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36724 (local address 127.0.98.1:46237)
0927 02:34:07.566985 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:07.566988 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:07.566999 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:34:07.567128 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:07.567131 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:07.567179 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:07.567259 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:34:07.567428 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:07.567902 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:07.568639 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:07.568832 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:07.568895 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:07.569131 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:34:07.569569 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:34:07.569657 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:07.569819 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 02:34:07.569879 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 02:34:07.572644 7543 raft_consensus.cc:1261] T fc76faad05a14bceb55e1848e6e01b35 P 733db7c7c5b54c6495ba5a80d1e50104 [term 1 FOLLOWER]: Refusing update from remote peer 48c360f547f84f9881723197501c1cc0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:34:07.572894 7680 raft_consensus.cc:1261] T fc76faad05a14bceb55e1848e6e01b35 P 65d9be8a776a46f6a53142ae0accd7ee [term 1 FOLLOWER]: Refusing update from remote peer 48c360f547f84f9881723197501c1cc0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:34:07.573230 7839 consensus_queue.cc:1035] T fc76faad05a14bceb55e1848e6e01b35 P 48c360f547f84f9881723197501c1cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "733db7c7c5b54c6495ba5a80d1e50104" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:34:07.573459 7837 consensus_queue.cc:1035] T fc76faad05a14bceb55e1848e6e01b35 P 48c360f547f84f9881723197501c1cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65d9be8a776a46f6a53142ae0accd7ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:34:07.575969 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:07.575394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.98.2:38667)
0927 02:34:07.575824 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:07.575885 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354}
I20260927 02:34:07.576272 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:07.575649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.98.3:33751)
0927 02:34:07.576153 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:07.576203 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427}
I20260927 02:34:07.579010 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:07.578681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.98.3:33777)
0927 02:34:07.578876 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:07.578932 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260927 02:34:07.579973 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:07.579670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.98.1:55337)
0927 02:34:07.579778 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:07.579841 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 02:34:07.582783 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:07.582374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.98.2:59763)
0927 02:34:07.582654 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:07.582707 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212}
I20260927 02:34:07.583137 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:07.581216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46055 (local address 127.0.98.1:48815)
0927 02:34:07.581910 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:07.581961 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":623}
I20260927 02:34:07.592351 7089 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:34:07.592535 7089 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:34:07.596598 7089 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 591735b7214747d5801020bb2bf64abd: Sending DeleteTablet for 3 replicas of tablet fc76faad05a14bceb55e1848e6e01b35
I20260927 02:34:07.597434 7386 tablet_service.cc:1514] Processing DeleteTablet for tablet fc76faad05a14bceb55e1848e6e01b35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:34:07 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:36718
I20260927 02:34:07.597426 7660 tablet_service.cc:1514] Processing DeleteTablet for tablet fc76faad05a14bceb55e1848e6e01b35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:34:07 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:43020
I20260927 02:34:07.597429 7523 tablet_service.cc:1514] Processing DeleteTablet for tablet fc76faad05a14bceb55e1848e6e01b35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:34:07 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:42586
I20260927 02:34:07.597826 7852 tablet_replica.cc:330] T fc76faad05a14bceb55e1848e6e01b35 P 65d9be8a776a46f6a53142ae0accd7ee: stopping tablet replica
I20260927 02:34:07.597988 7852 raft_consensus.cc:2229] T fc76faad05a14bceb55e1848e6e01b35 P 65d9be8a776a46f6a53142ae0accd7ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:34:07.598127 7852 raft_consensus.cc:2258] T fc76faad05a14bceb55e1848e6e01b35 P 65d9be8a776a46f6a53142ae0accd7ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:34:07.598358 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 7234
I20260927 02:34:07.598510 7854 tablet_replica.cc:330] T fc76faad05a14bceb55e1848e6e01b35 P 733db7c7c5b54c6495ba5a80d1e50104: stopping tablet replica
I20260927 02:34:07.599025 7854 raft_consensus.cc:2229] T fc76faad05a14bceb55e1848e6e01b35 P 733db7c7c5b54c6495ba5a80d1e50104 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:34:07.599246 7854 raft_consensus.cc:2258] T fc76faad05a14bceb55e1848e6e01b35 P 733db7c7c5b54c6495ba5a80d1e50104 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:34:07.600030 7854 ts_tablet_manager.cc:1905] T fc76faad05a14bceb55e1848e6e01b35 P 733db7c7c5b54c6495ba5a80d1e50104: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:34:07.601334 7852 ts_tablet_manager.cc:1905] T fc76faad05a14bceb55e1848e6e01b35 P 65d9be8a776a46f6a53142ae0accd7ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:34:07.602382 7854 ts_tablet_manager.cc:1918] T fc76faad05a14bceb55e1848e6e01b35 P 733db7c7c5b54c6495ba5a80d1e50104: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:34:07.602463 7854 log.cc:1196] T fc76faad05a14bceb55e1848e6e01b35 P 733db7c7c5b54c6495ba5a80d1e50104: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-1/wal/wals/fc76faad05a14bceb55e1848e6e01b35
I20260927 02:34:07.602694 7854 ts_tablet_manager.cc:1939] T fc76faad05a14bceb55e1848e6e01b35 P 733db7c7c5b54c6495ba5a80d1e50104: Deleting consensus metadata
I20260927 02:34:07.603556 7074 catalog_manager.cc:4841] TS 733db7c7c5b54c6495ba5a80d1e50104 (127.0.98.2:37325): tablet fc76faad05a14bceb55e1848e6e01b35 (table test-table [id=d1110b75e20f44caa7f6ce07af18890e]) successfully deleted
I20260927 02:34:07.603921 7852 ts_tablet_manager.cc:1918] T fc76faad05a14bceb55e1848e6e01b35 P 65d9be8a776a46f6a53142ae0accd7ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:34:07.603997 7852 log.cc:1196] T fc76faad05a14bceb55e1848e6e01b35 P 65d9be8a776a46f6a53142ae0accd7ee: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476410933508-392-0/minicluster-data/ts-2/wal/wals/fc76faad05a14bceb55e1848e6e01b35
I20260927 02:34:07.604249 7852 ts_tablet_manager.cc:1939] T fc76faad05a14bceb55e1848e6e01b35 P 65d9be8a776a46f6a53142ae0accd7ee: Deleting consensus metadata
I20260927 02:34:07.605116 7076 catalog_manager.cc:4841] TS 65d9be8a776a46f6a53142ae0accd7ee (127.0.98.3:43921): tablet fc76faad05a14bceb55e1848e6e01b35 (table test-table [id=d1110b75e20f44caa7f6ce07af18890e]) successfully deleted
I20260927 02:34:07.605775 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 7460
I20260927 02:34:07.605896 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:07.605630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46237 (local address 127.0.0.1:36728)
0927 02:34:07.605761 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:07.605813 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 02:34:07.606462 7073 catalog_manager.cc:4584] TS 48c360f547f84f9881723197501c1cc0 (127.0.98.1:46237): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fc76faad05a14bceb55e1848e6e01b35: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46237: connect: Connection refused (error 111)
I20260927 02:34:07.610831 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 7596
I20260927 02:34:07.616106 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 6953
I20260927 02:34:07.620353 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 7061
2026-09-27T02:34:07Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6693 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:34:07 dist-test-slave-grlk krb5kdc[7857](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:34:07 dist-test-slave-grlk krb5kdc[7857](info): set up 2 sockets
Sep 27 02:34:07 dist-test-slave-grlk krb5kdc[7857](info): commencing operation
krb5kdc: starting...
W20260927 02:34:08.577353 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/krb5kdc/test-user.keytab.
Sep 27 02:34:08 dist-test-slave-grlk krb5kdc[7857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:34:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:34:08Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 02:34:08.692111 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33027
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:34:08.802940 7870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:34:08.803226 7870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:34:08.803288 7870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:34:08.807235 7870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:34:08.807288 7870 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 02:34:08.807304 7870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:34:08.807320 7870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:34:08.807335 7870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:34:08.807354 7870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:34:08.812566 7870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33027
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:34:08.813555 7870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:34:08.814509 7870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:34:08.822744 7878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:08.822775 7876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:08.822855 7875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:08.822894 7870 server_base.cc:1034] running on GCE node
I20260927 02:34:08.823560 7870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:34:08.824647 7870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:34:08.825798 7870 hybrid_clock.cc:648] HybridClock initialized: now 1790476448825766 us; error 48 us; skew 500 ppm
Sep 27 02:34:08 dist-test-slave-grlk krb5kdc[7857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476448, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:34:08.828491 7870 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 02:34:08.829368 7870 webserver.cc:460] Webserver started at http://127.0.98.62:41483/ using document root <none> and password file <none>
I20260927 02:34:08.829865 7870 fs_manager.cc:359] Metadata directory not provided
I20260927 02:34:08.829918 7870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:34:08.830073 7870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:34:08.831992 7870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "e302084e70d14287803dacb09876350e"
format_stamp: "Formatted at 2026-09-27 02:34:08 on dist-test-slave-grlk"
server_key: "6d3c7d21c2b3fbba63dd45dc36c8bd04"
server_key_iv: "66d7d1c1299b559d495440b903cd92bb"
server_key_version: "encryptionkey@0"
I20260927 02:34:08.832479 7870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "e302084e70d14287803dacb09876350e"
format_stamp: "Formatted at 2026-09-27 02:34:08 on dist-test-slave-grlk"
server_key: "6d3c7d21c2b3fbba63dd45dc36c8bd04"
server_key_iv: "66d7d1c1299b559d495440b903cd92bb"
server_key_version: "encryptionkey@0"
I20260927 02:34:08.836007 7870 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 02:34:08.838366 7885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:08.839545 7870 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:34:08.839645 7870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "e302084e70d14287803dacb09876350e"
format_stamp: "Formatted at 2026-09-27 02:34:08 on dist-test-slave-grlk"
server_key: "6d3c7d21c2b3fbba63dd45dc36c8bd04"
server_key_iv: "66d7d1c1299b559d495440b903cd92bb"
server_key_version: "encryptionkey@0"
I20260927 02:34:08.839733 7870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:34:08.858323 7870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:34:08.862125 7870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:34:08.862298 7870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:34:08.870714 7870 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33027
I20260927 02:34:08.870714 7947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33027 every 8 connection(s)
I20260927 02:34:08.871718 7870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:34:08.874528 7948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:34:08.879345 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 7870
I20260927 02:34:08.879490 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:34:08.879783 392 external_mini_cluster.cc:1428] Setting key 4716570be899d19049f76ff61ce2972e
I20260927 02:34:08.880213 7948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e302084e70d14287803dacb09876350e: Bootstrap starting.
I20260927 02:34:08.882653 7948 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e302084e70d14287803dacb09876350e: Neither blocks nor log segments found. Creating new log.
I20260927 02:34:08.883502 7948 log.cc:824] T 00000000000000000000000000000000 P e302084e70d14287803dacb09876350e: Log is configured to *not* fsync() on all Append() calls
I20260927 02:34:08.885437 7948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e302084e70d14287803dacb09876350e: No bootstrap required, opened a new log
Sep 27 02:34:08 dist-test-slave-grlk krb5kdc[7857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:34:08.888494 7948 raft_consensus.cc:348] T 00000000000000000000000000000000 P e302084e70d14287803dacb09876350e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e302084e70d14287803dacb09876350e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33027 } }
I20260927 02:34:08.888723 7948 raft_consensus.cc:374] T 00000000000000000000000000000000 P e302084e70d14287803dacb09876350e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:34:08.888780 7948 raft_consensus.cc:729] T 00000000000000000000000000000000 P e302084e70d14287803dacb09876350e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e302084e70d14287803dacb09876350e, State: Initialized, Role: FOLLOWER
I20260927 02:34:08.889256 7948 consensus_queue.cc:260] T 00000000000000000000000000000000 P e302084e70d14287803dacb09876350e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e302084e70d14287803dacb09876350e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33027 } }
I20260927 02:34:08.889385 7948 raft_consensus.cc:388] T 00000000000000000000000000000000 P e302084e70d14287803dacb09876350e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:34:08.889451 7948 raft_consensus.cc:482] T 00000000000000000000000000000000 P e302084e70d14287803dacb09876350e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:34:08.889544 7948 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e302084e70d14287803dacb09876350e [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:34:08.890575 7948 raft_consensus.cc:504] T 00000000000000000000000000000000 P e302084e70d14287803dacb09876350e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e302084e70d14287803dacb09876350e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33027 } }
I20260927 02:34:08.890929 7948 leader_election.cc:302] T 00000000000000000000000000000000 P e302084e70d14287803dacb09876350e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e302084e70d14287803dacb09876350e; no voters:
I20260927 02:34:08.891243 7948 leader_election.cc:288] T 00000000000000000000000000000000 P e302084e70d14287803dacb09876350e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:34:08.891314 7953 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e302084e70d14287803dacb09876350e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:34:08.891532 7953 raft_consensus.cc:686] T 00000000000000000000000000000000 P e302084e70d14287803dacb09876350e [term 1 LEADER]: Becoming Leader. State: Replica: e302084e70d14287803dacb09876350e, State: Running, Role: LEADER
I20260927 02:34:08.891865 7953 consensus_queue.cc:237] T 00000000000000000000000000000000 P e302084e70d14287803dacb09876350e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e302084e70d14287803dacb09876350e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33027 } }
I20260927 02:34:08.892565 7948 sys_catalog.cc:564] T 00000000000000000000000000000000 P e302084e70d14287803dacb09876350e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:34:08.893528 7954 sys_catalog.cc:455] T 00000000000000000000000000000000 P e302084e70d14287803dacb09876350e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e302084e70d14287803dacb09876350e. Latest consensus state: current_term: 1 leader_uuid: "e302084e70d14287803dacb09876350e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e302084e70d14287803dacb09876350e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33027 } } }
I20260927 02:34:08.893648 7954 sys_catalog.cc:458] T 00000000000000000000000000000000 P e302084e70d14287803dacb09876350e [sys.catalog]: This master's current role is: LEADER
I20260927 02:34:08.893889 7951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:08.881274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35052 (local address 127.0.98.62:33027)
0927 02:34:08.881662 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:08.881676 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:08.881710 (+ 34us) server_negotiation.cc:407] Connection header received
0927 02:34:08.882321 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:08.882337 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:08.882666 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:08.882951 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:08.883772 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:08.884848 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:08.885950 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:08.886387 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:08.886510 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:08.888894 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:08.888899 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:08.888912 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:08.888963 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:08.890793 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:08.891195 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:08.891200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:08.891206 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:08.891340 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:08.891661 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:08.891664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:08.891666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:08.892126 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:08.892308 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:08.892584 (+ 276us) server_negotiation.cc:299] Negotiation successful
0927 02:34:08.892876 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":118,"threads_started":1}
I20260927 02:34:08.893954 7955 sys_catalog.cc:455] T 00000000000000000000000000000000 P e302084e70d14287803dacb09876350e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e302084e70d14287803dacb09876350e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e302084e70d14287803dacb09876350e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33027 } } }
I20260927 02:34:08.894141 7955 sys_catalog.cc:458] T 00000000000000000000000000000000 P e302084e70d14287803dacb09876350e [sys.catalog]: This master's current role is: LEADER
W20260927 02:34:08.897552 7968 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e302084e70d14287803dacb09876350e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:34:08.897615 7968 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:34:08.897697 7962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:34:08.898594 7962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:34:08.904503 7962 catalog_manager.cc:1347] Generated new cluster ID: 3a1a66213c30490894cdc51e735b7ece
I20260927 02:34:08.904572 7962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:34:08.913594 7962 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:34:08.914870 7962 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:34:08.921819 7962 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e302084e70d14287803dacb09876350e: Generated new TSK 0
I20260927 02:34:08.922624 7962 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 02:34:08.952059 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33027
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:34:09.061466 7974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:34:09.061753 7974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:34:09.061812 7974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:34:09.065881 7974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:34:09.065943 7974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:34:09.065982 7974 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:34:09.066017 7974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:34:09.072712 7974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:34:09.073874 7974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:34:09.074887 7974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:34:09.084393 7980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:09.084411 7979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:09.084388 7982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:09.084710 7974 server_base.cc:1034] running on GCE node
I20260927 02:34:09.085119 7974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:34:09.085731 7974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:34:09.086879 7974 hybrid_clock.cc:648] HybridClock initialized: now 1790476449086851 us; error 42 us; skew 500 ppm
Sep 27 02:34:09 dist-test-slave-grlk krb5kdc[7857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476449, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:34:09.089468 7974 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 02:34:09.090329 7974 webserver.cc:460] Webserver started at http://127.0.98.1:35447/ using document root <none> and password file <none>
I20260927 02:34:09.090844 7974 fs_manager.cc:359] Metadata directory not provided
I20260927 02:34:09.090902 7974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:34:09.091089 7974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:34:09.093197 7974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-0/data/instance:
uuid: "dfcc410419d7462193cfcc270fc06b54"
format_stamp: "Formatted at 2026-09-27 02:34:09 on dist-test-slave-grlk"
server_key: "27c6338e5810eb8a205222e7a2b66546"
server_key_iv: "127b79a33ed18fc77f4754c90c6d0db8"
server_key_version: "encryptionkey@0"
I20260927 02:34:09.093737 7974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-0/wal/instance:
uuid: "dfcc410419d7462193cfcc270fc06b54"
format_stamp: "Formatted at 2026-09-27 02:34:09 on dist-test-slave-grlk"
server_key: "27c6338e5810eb8a205222e7a2b66546"
server_key_iv: "127b79a33ed18fc77f4754c90c6d0db8"
server_key_version: "encryptionkey@0"
I20260927 02:34:09.097224 7974 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:34:09.099469 7989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:09.100602 7974 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:34:09.100701 7974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-0/wal
uuid: "dfcc410419d7462193cfcc270fc06b54"
format_stamp: "Formatted at 2026-09-27 02:34:09 on dist-test-slave-grlk"
server_key: "27c6338e5810eb8a205222e7a2b66546"
server_key_iv: "127b79a33ed18fc77f4754c90c6d0db8"
server_key_version: "encryptionkey@0"
I20260927 02:34:09.100796 7974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:34:09.133430 7974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:34:09.136353 7974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:34:09.136533 7974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:34:09.138025 7974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:34:09.138075 7974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:09.138130 7974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:34:09.138191 7974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 02:34:09 dist-test-slave-grlk krb5kdc[7857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476449, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:34:09.148231 7974 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35293
I20260927 02:34:09.148389 8104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35293 every 8 connection(s)
I20260927 02:34:09.149197 7974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:34:09.151237 7951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:09.138904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51969 (local address 127.0.98.62:33027)
0927 02:34:09.139015 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:09.139019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:09.140048 (+ 1029us) server_negotiation.cc:407] Connection header received
0927 02:34:09.141102 (+ 1054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:09.141106 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:09.141165 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:09.141228 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:09.142579 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.143401 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.144835 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.145188 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.145282 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:09.147771 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:09.147774 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:09.147776 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:09.147802 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:09.149378 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:09.149943 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:09.149947 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:09.149948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:09.150036 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:09.150356 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:09.150359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:09.150360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:09.150532 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:09.150602 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:09.151007 (+ 405us) server_negotiation.cc:299] Negotiation successful
0927 02:34:09.151108 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 02:34:09.151989 7998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:09.139417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33027 (local address 127.0.98.1:51969)
0927 02:34:09.139899 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:09.139942 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:09.140841 (+ 899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:09.141548 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:09.141554 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:09.141876 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:09.142417 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:09.142459 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.143530 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.143534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:09.144720 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:09.144730 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.145353 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.145357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:09.145523 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:09.146095 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:09.146111 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:09.147596 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:34:09.149481 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:09.149486 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:09.149501 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:09.149829 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:09.150136 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:09.150138 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:09.150140 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:09.150278 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:09.150604 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:34:09.150609 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:34:09.150823 (+ 214us) client_negotiation.cc:769] Sending connection context
0927 02:34:09.150973 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 02:34:09.151230 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":142,"threads_started":1}
I20260927 02:34:09.153220 8105 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33027
I20260927 02:34:09.153352 8105 heartbeater.cc:464] Registering TS with master...
I20260927 02:34:09.153822 8105 heartbeater.cc:511] Master 127.0.98.62:33027 requested a full tablet report, sending...
I20260927 02:34:09.155251 7902 ts_manager.cc:194] Registered new tserver with Master: dfcc410419d7462193cfcc270fc06b54 (127.0.98.1:35293)
I20260927 02:34:09.156302 7902 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51969
I20260927 02:34:09.158792 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 7974
I20260927 02:34:09.158908 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:34:09.159158 392 external_mini_cluster.cc:1428] Setting key 0dec19a4723ac1a00a7808cd889c4f6c
I20260927 02:34:09.163337 8111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:09.155143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33027 (local address 127.0.98.1:54649)
0927 02:34:09.155388 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:09.155401 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:09.155480 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:09.157070 (+ 1590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:09.157073 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:09.157253 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:09.157351 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:09.157357 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.158279 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.158281 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:09.158828 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:09.158835 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.159309 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.159313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:09.159406 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:09.159909 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:09.159920 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:09.160293 (+ 373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:34:09.161992 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:09.161996 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:09.161998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:09.162247 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:09.162475 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:09.162477 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:09.162480 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:09.162562 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:09.162857 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:34:09.162860 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:34:09.162952 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 02:34:09.163051 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 02:34:09.163171 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":82,"threads_started":1}
I20260927 02:34:09.163340 7951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:09.155240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54649 (local address 127.0.98.62:33027)
0927 02:34:09.156828 (+ 1588us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:09.156832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:09.156842 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:34:09.156895 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:09.156897 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:09.156939 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:09.157033 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:09.157459 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.158176 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.158936 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.159199 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.159294 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:09.160406 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:09.160410 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:09.160412 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:09.160435 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:09.161884 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:09.162333 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:09.162335 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:09.162337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:09.162402 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:09.162632 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:09.162633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:09.162635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:09.162778 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:09.162844 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:09.163094 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 02:34:09.163212 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1533}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 02:34:09.183207 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33027
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:34:09.293553 8114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:34:09.293831 8114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:34:09.293874 8114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:34:09.297888 8114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:34:09.297948 8114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:34:09.297984 8114 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:34:09.298014 8114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:34:09.303100 8114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:34:09.304090 8114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:34:09.305121 8114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:34:09.314118 8119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:09.314275 8114 server_base.cc:1034] running on GCE node
W20260927 02:34:09.314208 8120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:09.314126 8122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:09.314731 8114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:34:09.315307 8114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:34:09.316452 8114 hybrid_clock.cc:648] HybridClock initialized: now 1790476449316438 us; error 27 us; skew 500 ppm
Sep 27 02:34:09 dist-test-slave-grlk krb5kdc[7857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476449, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:34:09.319087 8114 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 02:34:09.319852 8114 webserver.cc:460] Webserver started at http://127.0.98.2:34185/ using document root <none> and password file <none>
I20260927 02:34:09.320365 8114 fs_manager.cc:359] Metadata directory not provided
I20260927 02:34:09.320427 8114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:34:09.320604 8114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:34:09.322541 8114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-1/data/instance:
uuid: "619d0542fa744235954689b0f8badebb"
format_stamp: "Formatted at 2026-09-27 02:34:09 on dist-test-slave-grlk"
server_key: "186f931c18180847891683b0726bdd2d"
server_key_iv: "403ea28b89cad8c20b8511b1fe133e3a"
server_key_version: "encryptionkey@0"
I20260927 02:34:09.323051 8114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-1/wal/instance:
uuid: "619d0542fa744235954689b0f8badebb"
format_stamp: "Formatted at 2026-09-27 02:34:09 on dist-test-slave-grlk"
server_key: "186f931c18180847891683b0726bdd2d"
server_key_iv: "403ea28b89cad8c20b8511b1fe133e3a"
server_key_version: "encryptionkey@0"
I20260927 02:34:09.326421 8114 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 02:34:09.328630 8129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:09.329742 8114 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:34:09.329845 8114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-1/wal
uuid: "619d0542fa744235954689b0f8badebb"
format_stamp: "Formatted at 2026-09-27 02:34:09 on dist-test-slave-grlk"
server_key: "186f931c18180847891683b0726bdd2d"
server_key_iv: "403ea28b89cad8c20b8511b1fe133e3a"
server_key_version: "encryptionkey@0"
I20260927 02:34:09.329933 8114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:34:09.357254 8114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:34:09.360350 8114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:34:09.360524 8114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:34:09.361943 8114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:34:09.361990 8114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:09.362049 8114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:34:09.362077 8114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 02:34:09 dist-test-slave-grlk krb5kdc[7857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476449, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:34:09.371981 8114 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43319
I20260927 02:34:09.372588 8244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43319 every 8 connection(s)
I20260927 02:34:09.372869 8114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:34:09.374397 7951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:09.362813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34945 (local address 127.0.98.62:33027)
0927 02:34:09.362924 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:09.362928 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:09.363899 (+ 971us) server_negotiation.cc:407] Connection header received
0927 02:34:09.364986 (+ 1087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:09.364990 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:09.365051 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:09.365168 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:09.366341 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.366960 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.368091 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.368327 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.368418 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:09.371091 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:09.371094 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:09.371096 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:09.371118 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:09.372428 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:09.373006 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:09.373009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:09.373011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:09.373082 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:09.373449 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:09.373452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:09.373454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:09.373685 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:09.373780 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:09.374176 (+ 396us) server_negotiation.cc:299] Negotiation successful
0927 02:34:09.374268 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 02:34:09.375092 8138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:09.363272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33027 (local address 127.0.98.2:34945)
0927 02:34:09.363751 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:09.363790 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:09.364710 (+ 920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:09.365347 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:09.365355 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:09.365714 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:09.366208 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:09.366222 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.367072 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.367075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:09.367979 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:09.367988 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.368468 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.368472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:09.368613 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:09.369196 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:09.369214 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:09.370943 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:34:09.372541 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:09.372546 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:09.372558 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:09.372874 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:09.373171 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:09.373174 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:09.373176 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:09.373337 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:09.373794 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:34:09.373799 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:34:09.373993 (+ 194us) client_negotiation.cc:769] Sending connection context
0927 02:34:09.374138 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 02:34:09.374357 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":140,"threads_started":1}
I20260927 02:34:09.376148 8245 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33027
I20260927 02:34:09.376281 8245 heartbeater.cc:464] Registering TS with master...
I20260927 02:34:09.376760 8245 heartbeater.cc:511] Master 127.0.98.62:33027 requested a full tablet report, sending...
I20260927 02:34:09.377763 7901 ts_manager.cc:194] Registered new tserver with Master: 619d0542fa744235954689b0f8badebb (127.0.98.2:43319)
I20260927 02:34:09.378473 7901 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34945
I20260927 02:34:09.379026 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 8114
I20260927 02:34:09.379155 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:34:09.379416 392 external_mini_cluster.cc:1428] Setting key 3245b9363232226da33ca99a5841f707
I20260927 02:34:09.384908 8251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:09.378175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33027 (local address 127.0.98.2:34987)
0927 02:34:09.378508 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:09.378519 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:09.378600 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:09.378805 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:09.378807 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:09.378985 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:09.379092 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:09.379099 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.380083 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.380086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:09.380738 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:09.380746 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.381197 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.381201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:09.381286 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:09.381688 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:09.381698 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:09.381972 (+ 274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:34:09.383507 (+ 1535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:09.383510 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:09.383512 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:09.383691 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:09.383968 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:09.383970 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:09.383972 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:09.384079 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:09.384513 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:34:09.384515 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:34:09.384590 (+ 75us) client_negotiation.cc:769] Sending connection context
0927 02:34:09.384670 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 02:34:09.384770 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":93,"threads_started":1}
I20260927 02:34:09.385039 7951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:09.378362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34987 (local address 127.0.98.62:33027)
0927 02:34:09.378457 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:09.378461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:09.378588 (+ 127us) server_negotiation.cc:407] Connection header received
0927 02:34:09.378679 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:09.378682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:09.378719 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:09.378778 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:09.379230 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.379934 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.380845 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.381095 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.381179 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:09.382102 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:09.382106 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:09.382108 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:09.382130 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:09.383389 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:09.383777 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:09.383781 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:09.383782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:09.383867 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:09.384164 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:09.384167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:09.384168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:09.384429 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:09.384528 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:09.384746 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 02:34:09.384850 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 02:34:09.404242 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33027
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:34:09.510106 8254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:34:09.510353 8254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:34:09.510396 8254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:34:09.514446 8254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:34:09.514516 8254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:34:09.514554 8254 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:34:09.514588 8254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:34:09.519734 8254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:34:09.520742 8254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:34:09.521726 8254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:34:09.530679 8262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:09.530649 8259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:09.530668 8260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:09.530817 8254 server_base.cc:1034] running on GCE node
I20260927 02:34:09.531294 8254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:34:09.531877 8254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:34:09.532989 8254 hybrid_clock.cc:648] HybridClock initialized: now 1790476449532962 us; error 34 us; skew 500 ppm
Sep 27 02:34:09 dist-test-slave-grlk krb5kdc[7857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476449, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:34:09.535578 8254 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 02:34:09.536307 8254 webserver.cc:460] Webserver started at http://127.0.98.3:43927/ using document root <none> and password file <none>
I20260927 02:34:09.536783 8254 fs_manager.cc:359] Metadata directory not provided
I20260927 02:34:09.536836 8254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:34:09.537027 8254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:34:09.538913 8254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-2/data/instance:
uuid: "cc8fdd96bba24e3682c3ba405fcc0a45"
format_stamp: "Formatted at 2026-09-27 02:34:09 on dist-test-slave-grlk"
server_key: "759722dc546be293147d2fae9a100d3e"
server_key_iv: "90f1e051a92c45408d560b60cb9fd763"
server_key_version: "encryptionkey@0"
I20260927 02:34:09.539384 8254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-2/wal/instance:
uuid: "cc8fdd96bba24e3682c3ba405fcc0a45"
format_stamp: "Formatted at 2026-09-27 02:34:09 on dist-test-slave-grlk"
server_key: "759722dc546be293147d2fae9a100d3e"
server_key_iv: "90f1e051a92c45408d560b60cb9fd763"
server_key_version: "encryptionkey@0"
I20260927 02:34:09.542773 8254 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:34:09.545110 8269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:09.546244 8254 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:34:09.546336 8254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-2/wal
uuid: "cc8fdd96bba24e3682c3ba405fcc0a45"
format_stamp: "Formatted at 2026-09-27 02:34:09 on dist-test-slave-grlk"
server_key: "759722dc546be293147d2fae9a100d3e"
server_key_iv: "90f1e051a92c45408d560b60cb9fd763"
server_key_version: "encryptionkey@0"
I20260927 02:34:09.546435 8254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:34:09.571555 8254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:34:09.574599 8254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:34:09.574769 8254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:34:09.576122 8254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:34:09.576174 8254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:09.576224 8254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:34:09.576252 8254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:09.585508 8254 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45319
Sep 27 02:34:09 dist-test-slave-grlk krb5kdc[7857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476449, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:34:09.585567 8384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45319 every 8 connection(s)
I20260927 02:34:09.586341 8254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:34:09.589237 7951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:09.576978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59945 (local address 127.0.98.62:33027)
0927 02:34:09.577112 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:09.577117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:09.577945 (+ 828us) server_negotiation.cc:407] Connection header received
0927 02:34:09.579163 (+ 1218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:09.579166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:09.579223 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:09.579279 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:09.580949 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.581504 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.583096 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.583315 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.583402 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:09.586167 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:09.586171 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:09.586173 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:09.586201 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:09.587633 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:09.588048 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:09.588050 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:09.588051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:09.588120 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:09.588372 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:09.588374 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:09.588375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:09.588528 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:09.588613 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:09.589031 (+ 418us) server_negotiation.cc:299] Negotiation successful
0927 02:34:09.589124 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 02:34:09.590018 8278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:09.577348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33027 (local address 127.0.98.3:59945)
0927 02:34:09.577791 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:09.577837 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:09.578924 (+ 1087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:09.579698 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:09.579705 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:09.580130 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:09.580789 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:09.580810 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.581634 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.581637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:09.582977 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:09.582987 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.583458 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.583462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:09.583659 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:09.584325 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:09.584348 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:09.586004 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:34:09.587727 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:09.587731 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:09.587742 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:09.587967 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:09.588184 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:09.588186 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:09.588187 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:09.588304 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:09.588613 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:34:09.588618 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:34:09.588823 (+ 205us) client_negotiation.cc:769] Sending connection context
0927 02:34:09.588996 (+ 173us) client_negotiation.cc:240] Negotiation successful
0927 02:34:09.589237 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":107,"threads_started":1}
I20260927 02:34:09.590008 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 8254
I20260927 02:34:09.590129 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:34:09.590395 392 external_mini_cluster.cc:1428] Setting key 5fbd08f67e41c8b93e570584b03a2714
I20260927 02:34:09.591190 8385 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33027
I20260927 02:34:09.591317 8385 heartbeater.cc:464] Registering TS with master...
I20260927 02:34:09.591831 8385 heartbeater.cc:511] Master 127.0.98.62:33027 requested a full tablet report, sending...
I20260927 02:34:09.592958 7902 ts_manager.cc:194] Registered new tserver with Master: cc8fdd96bba24e3682c3ba405fcc0a45 (127.0.98.3:45319)
I20260927 02:34:09.593719 7902 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59945
I20260927 02:34:09.599295 7951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:09.593417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43091 (local address 127.0.98.62:33027)
0927 02:34:09.593520 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:09.593524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:09.593694 (+ 170us) server_negotiation.cc:407] Connection header received
0927 02:34:09.593827 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:09.593830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:09.593870 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:09.593932 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:09.594365 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.594844 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.595801 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.596003 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.596071 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:09.597041 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:09.597043 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:09.597046 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:09.597062 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:09.598119 (+ 1057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:09.598446 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:09.598449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:09.598450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:09.598512 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:09.598704 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:09.598706 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:09.598707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:09.598841 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:09.598905 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:09.599113 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 02:34:09.599192 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 02:34:09.599325 8391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:09.593373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33027 (local address 127.0.98.3:43091)
0927 02:34:09.593630 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:09.593644 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:09.593743 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:09.593957 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:09.593960 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:09.594143 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:09.594277 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:09.594284 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.594963 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.594967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:09.595712 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:09.595721 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.596093 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.596096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:09.596162 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:09.596604 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:09.596617 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:09.596900 (+ 283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:34:09.598206 (+ 1306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:09.598208 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:09.598210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:09.598377 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:09.598579 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:09.598581 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:09.598582 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:09.598644 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:09.598900 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:34:09.598902 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:34:09.598984 (+ 82us) client_negotiation.cc:769] Sending connection context
0927 02:34:09.599080 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 02:34:09.599203 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":93,"threads_started":1}
I20260927 02:34:09.603555 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 02:34:09 dist-test-slave-grlk krb5kdc[7857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476449, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 02:34:09 dist-test-slave-grlk krb5kdc[7857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:09 dist-test-slave-grlk krb5kdc[7857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 02:34:09.624467 7951 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:09.618926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35062 (local address 127.0.98.62:33027)
0927 02:34:09.619041 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:09.619044 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:09.619055 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:34:09.619092 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:09.619094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:09.619130 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:09.619195 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:09.619862 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.620368 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.621205 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.621414 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.621495 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:09.624355 (+ 2860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35062: BlockingRecv error: recv got EOF from 127.0.0.1:35062 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 27 02:34:09 dist-test-slave-grlk krb5kdc[7857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476449, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:34:09.633399 7951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:09.625880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35068 (local address 127.0.98.62:33027)
0927 02:34:09.625986 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:09.625990 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:09.626050 (+ 60us) server_negotiation.cc:407] Connection header received
0927 02:34:09.626175 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:09.626178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:09.626218 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:09.626281 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:09.626831 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.627313 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.628148 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.628353 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.628437 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:09.630604 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:09.630607 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:09.630609 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:09.630631 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:09.631853 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:09.632243 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:09.632247 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:09.632248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:09.632317 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:09.632553 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:09.632564 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:09.632565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:09.632758 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:09.632890 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:09.633143 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 02:34:09.633247 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 02:34:09.636075 7902 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35068:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:34:09.639413 7902 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:34:09.653158 8407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:09.647498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43319 (local address 127.0.0.1:41258)
0927 02:34:09.648196 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:09.648229 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:09.648349 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:09.648865 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:09.648870 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:09.648889 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:34:09.649048 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:09.649054 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.650332 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.650336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:09.651613 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:09.651623 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.652589 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.652593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:09.652699 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:09.652736 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 02:34:09.652838 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 02:34:09.652997 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"mutex_wait_us":1,"thread_start_us":58,"threads_started":1}
I20260927 02:34:09.653175 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:09.647688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41258 (local address 127.0.98.2:43319)
0927 02:34:09.648075 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:09.648086 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:09.648318 (+ 232us) server_negotiation.cc:407] Connection header received
0927 02:34:09.648490 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:09.648496 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:09.648708 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:09.649184 (+ 476us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:34:09.649308 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.650190 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.651746 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.652495 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.652591 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:09.652744 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:09.652872 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 02:34:09.652990 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":100,"threads_started":1}
I20260927 02:34:09.653700 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:09.647840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35293 (local address 127.0.0.1:60972)
0927 02:34:09.648798 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:09.648810 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:09.648879 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:09.649522 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:09.649525 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:09.649542 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:34:09.649644 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:09.649650 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.650977 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.650980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:09.652078 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:09.652086 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.653339 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.653343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:09.653424 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:09.653443 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 02:34:09.653530 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 02:34:09.653592 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":904,"thread_start_us":76,"threads_started":1}
I20260927 02:34:09.653764 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:09.648202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34830 (local address 127.0.98.3:45319)
0927 02:34:09.648876 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:09.648886 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:09.648907 (+ 21us) server_negotiation.cc:407] Connection header received
0927 02:34:09.649015 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:09.649020 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:09.649236 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:09.649377 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:34:09.649555 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.650686 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.652000 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.652723 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.652821 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:09.652999 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:09.653538 (+ 539us) server_negotiation.cc:299] Negotiation successful
0927 02:34:09.653622 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":87,"threads_started":1}
I20260927 02:34:09.653807 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:09.648202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60972 (local address 127.0.98.1:35293)
0927 02:34:09.648688 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:09.648703 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:09.648974 (+ 271us) server_negotiation.cc:407] Connection header received
0927 02:34:09.649047 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:09.649052 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:09.649263 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:09.649397 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:34:09.649761 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.650879 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.652191 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.653019 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.653169 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:09.653293 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:09.653578 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 02:34:09.653666 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":86,"threads_started":1}
I20260927 02:34:09.653448 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:09.647634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45319 (local address 127.0.0.1:34830)
0927 02:34:09.648568 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:09.648579 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:09.648647 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:09.649355 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:09.649358 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:09.649374 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:34:09.649468 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:09.649475 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.650808 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.650811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:09.651885 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:09.651894 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.653060 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.653064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:09.653161 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:09.653183 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 02:34:09.653244 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 02:34:09.653315 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":879,"mutex_wait_us":28,"thread_start_us":87,"threads_started":1}
I20260927 02:34:09.656180 8179 tablet_service.cc:1467] Processing CreateTablet for tablet a9edb14c698546b295595ccb5e23a7cf (DEFAULT_TABLE table=test-table [id=cecb7c8730fe4490b3f01c458c5073af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:34:09.656574 8039 tablet_service.cc:1467] Processing CreateTablet for tablet a9edb14c698546b295595ccb5e23a7cf (DEFAULT_TABLE table=test-table [id=cecb7c8730fe4490b3f01c458c5073af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:34:09.657119 8179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9edb14c698546b295595ccb5e23a7cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:34:09.656891 8319 tablet_service.cc:1467] Processing CreateTablet for tablet a9edb14c698546b295595ccb5e23a7cf (DEFAULT_TABLE table=test-table [id=cecb7c8730fe4490b3f01c458c5073af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:34:09.657578 8039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9edb14c698546b295595ccb5e23a7cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:34:09.657770 8319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9edb14c698546b295595ccb5e23a7cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:34:09.663841 8413 tablet_bootstrap.cc:492] T a9edb14c698546b295595ccb5e23a7cf P 619d0542fa744235954689b0f8badebb: Bootstrap starting.
I20260927 02:34:09.664614 8414 tablet_bootstrap.cc:492] T a9edb14c698546b295595ccb5e23a7cf P dfcc410419d7462193cfcc270fc06b54: Bootstrap starting.
I20260927 02:34:09.665239 8415 tablet_bootstrap.cc:492] T a9edb14c698546b295595ccb5e23a7cf P cc8fdd96bba24e3682c3ba405fcc0a45: Bootstrap starting.
I20260927 02:34:09.666077 8413 tablet_bootstrap.cc:654] T a9edb14c698546b295595ccb5e23a7cf P 619d0542fa744235954689b0f8badebb: Neither blocks nor log segments found. Creating new log.
I20260927 02:34:09.666590 8414 tablet_bootstrap.cc:654] T a9edb14c698546b295595ccb5e23a7cf P dfcc410419d7462193cfcc270fc06b54: Neither blocks nor log segments found. Creating new log.
I20260927 02:34:09.666903 8415 tablet_bootstrap.cc:654] T a9edb14c698546b295595ccb5e23a7cf P cc8fdd96bba24e3682c3ba405fcc0a45: Neither blocks nor log segments found. Creating new log.
I20260927 02:34:09.666909 8413 log.cc:824] T a9edb14c698546b295595ccb5e23a7cf P 619d0542fa744235954689b0f8badebb: Log is configured to *not* fsync() on all Append() calls
I20260927 02:34:09.667594 8415 log.cc:824] T a9edb14c698546b295595ccb5e23a7cf P cc8fdd96bba24e3682c3ba405fcc0a45: Log is configured to *not* fsync() on all Append() calls
I20260927 02:34:09.667688 8414 log.cc:824] T a9edb14c698546b295595ccb5e23a7cf P dfcc410419d7462193cfcc270fc06b54: Log is configured to *not* fsync() on all Append() calls
I20260927 02:34:09.669009 8413 tablet_bootstrap.cc:492] T a9edb14c698546b295595ccb5e23a7cf P 619d0542fa744235954689b0f8badebb: No bootstrap required, opened a new log
I20260927 02:34:09.669235 8413 ts_tablet_manager.cc:1397] T a9edb14c698546b295595ccb5e23a7cf P 619d0542fa744235954689b0f8badebb: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 02:34:09.669395 8415 tablet_bootstrap.cc:492] T a9edb14c698546b295595ccb5e23a7cf P cc8fdd96bba24e3682c3ba405fcc0a45: No bootstrap required, opened a new log
I20260927 02:34:09.669395 8414 tablet_bootstrap.cc:492] T a9edb14c698546b295595ccb5e23a7cf P dfcc410419d7462193cfcc270fc06b54: No bootstrap required, opened a new log
I20260927 02:34:09.669565 8414 ts_tablet_manager.cc:1397] T a9edb14c698546b295595ccb5e23a7cf P dfcc410419d7462193cfcc270fc06b54: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 02:34:09.669565 8415 ts_tablet_manager.cc:1397] T a9edb14c698546b295595ccb5e23a7cf P cc8fdd96bba24e3682c3ba405fcc0a45: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260927 02:34:09.672039 8413 raft_consensus.cc:348] T a9edb14c698546b295595ccb5e23a7cf P 619d0542fa744235954689b0f8badebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } } peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } }
I20260927 02:34:09.672243 8413 raft_consensus.cc:374] T a9edb14c698546b295595ccb5e23a7cf P 619d0542fa744235954689b0f8badebb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:34:09.672313 8413 raft_consensus.cc:729] T a9edb14c698546b295595ccb5e23a7cf P 619d0542fa744235954689b0f8badebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 619d0542fa744235954689b0f8badebb, State: Initialized, Role: FOLLOWER
I20260927 02:34:09.672729 8413 consensus_queue.cc:260] T a9edb14c698546b295595ccb5e23a7cf P 619d0542fa744235954689b0f8badebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } } peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } }
I20260927 02:34:09.673319 8245 heartbeater.cc:503] Master 127.0.98.62:33027 was elected leader, sending a full tablet report...
I20260927 02:34:09.673172 8415 raft_consensus.cc:348] T a9edb14c698546b295595ccb5e23a7cf P cc8fdd96bba24e3682c3ba405fcc0a45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } } peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } }
I20260927 02:34:09.673162 8414 raft_consensus.cc:348] T a9edb14c698546b295595ccb5e23a7cf P dfcc410419d7462193cfcc270fc06b54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } } peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } }
I20260927 02:34:09.673389 8415 raft_consensus.cc:374] T a9edb14c698546b295595ccb5e23a7cf P cc8fdd96bba24e3682c3ba405fcc0a45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:34:09.673389 8414 raft_consensus.cc:374] T a9edb14c698546b295595ccb5e23a7cf P dfcc410419d7462193cfcc270fc06b54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:34:09.673435 8415 raft_consensus.cc:729] T a9edb14c698546b295595ccb5e23a7cf P cc8fdd96bba24e3682c3ba405fcc0a45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8fdd96bba24e3682c3ba405fcc0a45, State: Initialized, Role: FOLLOWER
I20260927 02:34:09.673436 8414 raft_consensus.cc:729] T a9edb14c698546b295595ccb5e23a7cf P dfcc410419d7462193cfcc270fc06b54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfcc410419d7462193cfcc270fc06b54, State: Initialized, Role: FOLLOWER
I20260927 02:34:09.673671 8413 ts_tablet_manager.cc:1428] T a9edb14c698546b295595ccb5e23a7cf P 619d0542fa744235954689b0f8badebb: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260927 02:34:09.673835 8415 consensus_queue.cc:260] T a9edb14c698546b295595ccb5e23a7cf P cc8fdd96bba24e3682c3ba405fcc0a45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } } peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } }
I20260927 02:34:09.673862 8414 consensus_queue.cc:260] T a9edb14c698546b295595ccb5e23a7cf P dfcc410419d7462193cfcc270fc06b54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } } peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } }
I20260927 02:34:09.674391 8105 heartbeater.cc:503] Master 127.0.98.62:33027 was elected leader, sending a full tablet report...
I20260927 02:34:09.674685 8415 ts_tablet_manager.cc:1428] T a9edb14c698546b295595ccb5e23a7cf P cc8fdd96bba24e3682c3ba405fcc0a45: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 02:34:09.674705 8414 ts_tablet_manager.cc:1428] T a9edb14c698546b295595ccb5e23a7cf P dfcc410419d7462193cfcc270fc06b54: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 02:34:09.674983 8385 heartbeater.cc:503] Master 127.0.98.62:33027 was elected leader, sending a full tablet report...
I20260927 02:34:09.816411 8419 raft_consensus.cc:482] T a9edb14c698546b295595ccb5e23a7cf P 619d0542fa744235954689b0f8badebb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:34:09.816622 8419 raft_consensus.cc:504] T a9edb14c698546b295595ccb5e23a7cf P 619d0542fa744235954689b0f8badebb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } } peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } }
I20260927 02:34:09.817754 8419 leader_election.cc:288] T a9edb14c698546b295595ccb5e23a7cf P 619d0542fa744235954689b0f8badebb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc8fdd96bba24e3682c3ba405fcc0a45 (127.0.98.3:45319), dfcc410419d7462193cfcc270fc06b54 (127.0.98.1:35293)
I20260927 02:34:09.821478 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:09.818110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59337 (local address 127.0.98.1:35293)
0927 02:34:09.818225 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:09.818229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:09.818242 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:34:09.818393 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:09.818396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:09.818451 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:09.818533 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:34:09.818810 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.819368 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.820553 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.820985 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.821079 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:09.821145 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:09.821322 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 02:34:09.821384 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 02:34:09.821509 8138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:09.818013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35293 (local address 127.0.98.2:59337)
0927 02:34:09.818148 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:09.818164 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:09.818267 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:09.818564 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:09.818568 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:09.818587 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:34:09.818701 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:09.818707 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.819481 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.819483 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:09.820413 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:09.820423 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.821103 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.821107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:09.821207 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:09.821232 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 02:34:09.821303 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 02:34:09.821375 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 02:34:09.822028 8059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9edb14c698546b295595ccb5e23a7cf" candidate_uuid: "619d0542fa744235954689b0f8badebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfcc410419d7462193cfcc270fc06b54" is_pre_election: true
I20260927 02:34:09.822180 8422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:09.818013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45319 (local address 127.0.98.2:36073)
0927 02:34:09.818585 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:09.818598 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:09.818680 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:09.818888 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:09.818890 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:09.818905 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:34:09.819338 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:09.819344 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.820044 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.820047 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:09.820893 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:09.820900 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.821811 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.821815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:09.821902 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:09.821918 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 02:34:09.821978 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 02:34:09.822037 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"mutex_wait_us":239,"thread_start_us":133,"threads_started":1}
I20260927 02:34:09.822194 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:09.818126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36073 (local address 127.0.98.3:45319)
0927 02:34:09.818236 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:09.818239 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:09.818663 (+ 424us) server_negotiation.cc:407] Connection header received
0927 02:34:09.818754 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:09.818757 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:09.818805 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:09.818875 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:34:09.819463 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.819936 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.821116 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.821688 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.821785 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:09.821842 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:09.822013 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 02:34:09.822091 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 02:34:09.822327 8059 raft_consensus.cc:2454] T a9edb14c698546b295595ccb5e23a7cf P dfcc410419d7462193cfcc270fc06b54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 619d0542fa744235954689b0f8badebb in term 0.
I20260927 02:34:09.822769 8130 leader_election.cc:302] T a9edb14c698546b295595ccb5e23a7cf P 619d0542fa744235954689b0f8badebb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 619d0542fa744235954689b0f8badebb, dfcc410419d7462193cfcc270fc06b54; no voters:
I20260927 02:34:09.822778 8339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9edb14c698546b295595ccb5e23a7cf" candidate_uuid: "619d0542fa744235954689b0f8badebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" is_pre_election: true
I20260927 02:34:09.822996 8419 raft_consensus.cc:2781] T a9edb14c698546b295595ccb5e23a7cf P 619d0542fa744235954689b0f8badebb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:34:09.823006 8339 raft_consensus.cc:2454] T a9edb14c698546b295595ccb5e23a7cf P cc8fdd96bba24e3682c3ba405fcc0a45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 619d0542fa744235954689b0f8badebb in term 0.
I20260927 02:34:09.823086 8419 raft_consensus.cc:482] T a9edb14c698546b295595ccb5e23a7cf P 619d0542fa744235954689b0f8badebb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:34:09.823136 8419 raft_consensus.cc:3037] T a9edb14c698546b295595ccb5e23a7cf P 619d0542fa744235954689b0f8badebb [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:34:09.824105 8419 raft_consensus.cc:504] T a9edb14c698546b295595ccb5e23a7cf P 619d0542fa744235954689b0f8badebb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } } peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } }
I20260927 02:34:09.824381 8419 leader_election.cc:288] T a9edb14c698546b295595ccb5e23a7cf P 619d0542fa744235954689b0f8badebb [CANDIDATE]: Term 1 election: Requested vote from peers cc8fdd96bba24e3682c3ba405fcc0a45 (127.0.98.3:45319), dfcc410419d7462193cfcc270fc06b54 (127.0.98.1:35293)
I20260927 02:34:09.824709 8339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9edb14c698546b295595ccb5e23a7cf" candidate_uuid: "619d0542fa744235954689b0f8badebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45"
I20260927 02:34:09.824708 8059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9edb14c698546b295595ccb5e23a7cf" candidate_uuid: "619d0542fa744235954689b0f8badebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfcc410419d7462193cfcc270fc06b54"
I20260927 02:34:09.824815 8339 raft_consensus.cc:3037] T a9edb14c698546b295595ccb5e23a7cf P cc8fdd96bba24e3682c3ba405fcc0a45 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:34:09.824836 8059 raft_consensus.cc:3037] T a9edb14c698546b295595ccb5e23a7cf P dfcc410419d7462193cfcc270fc06b54 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:34:09.826033 8059 raft_consensus.cc:2454] T a9edb14c698546b295595ccb5e23a7cf P dfcc410419d7462193cfcc270fc06b54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 619d0542fa744235954689b0f8badebb in term 1.
I20260927 02:34:09.826228 8339 raft_consensus.cc:2454] T a9edb14c698546b295595ccb5e23a7cf P cc8fdd96bba24e3682c3ba405fcc0a45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 619d0542fa744235954689b0f8badebb in term 1.
I20260927 02:34:09.826339 8130 leader_election.cc:302] T a9edb14c698546b295595ccb5e23a7cf P 619d0542fa744235954689b0f8badebb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 619d0542fa744235954689b0f8badebb, dfcc410419d7462193cfcc270fc06b54; no voters:
I20260927 02:34:09.826493 8419 raft_consensus.cc:2781] T a9edb14c698546b295595ccb5e23a7cf P 619d0542fa744235954689b0f8badebb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:34:09.826676 8419 raft_consensus.cc:686] T a9edb14c698546b295595ccb5e23a7cf P 619d0542fa744235954689b0f8badebb [term 1 LEADER]: Becoming Leader. State: Replica: 619d0542fa744235954689b0f8badebb, State: Running, Role: LEADER
I20260927 02:34:09.826963 8419 consensus_queue.cc:237] T a9edb14c698546b295595ccb5e23a7cf P 619d0542fa744235954689b0f8badebb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } } peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } }
I20260927 02:34:09.829751 7901 catalog_manager.cc:5495] T a9edb14c698546b295595ccb5e23a7cf P 619d0542fa744235954689b0f8badebb reported cstate change: term changed from 0 to 1, leader changed from <none> to 619d0542fa744235954689b0f8badebb (127.0.98.2). New cstate: current_term: 1 leader_uuid: "619d0542fa744235954689b0f8badebb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } health_report { overall_health: UNKNOWN } } }
W20260927 02:34:09.837293 8386 tablet.cc:2369] T a9edb14c698546b295595ccb5e23a7cf P cc8fdd96bba24e3682c3ba405fcc0a45: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:34:09.840444 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:09.837166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41270 (local address 127.0.98.2:43319)
0927 02:34:09.837290 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:09.837295 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:09.837310 (+ 15us) server_negotiation.cc:407] Connection header received
0927 02:34:09.837404 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:09.837407 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:09.837471 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:09.837566 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:34:09.837776 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.838292 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.839166 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.839369 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.839435 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:09.839655 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:34:09.840068 (+ 413us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:34:09.840157 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:09.840268 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 02:34:09.840318 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 02:34:09.846042 8339 raft_consensus.cc:1261] T a9edb14c698546b295595ccb5e23a7cf P cc8fdd96bba24e3682c3ba405fcc0a45 [term 1 FOLLOWER]: Refusing update from remote peer 619d0542fa744235954689b0f8badebb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:34:09.846042 8059 raft_consensus.cc:1261] T a9edb14c698546b295595ccb5e23a7cf P dfcc410419d7462193cfcc270fc06b54 [term 1 FOLLOWER]: Refusing update from remote peer 619d0542fa744235954689b0f8badebb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:34:09.846746 8423 consensus_queue.cc:1035] T a9edb14c698546b295595ccb5e23a7cf P 619d0542fa744235954689b0f8badebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:34:09.846956 8419 consensus_queue.cc:1035] T a9edb14c698546b295595ccb5e23a7cf P 619d0542fa744235954689b0f8badebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:34:09.854415 8427 mvcc.cc:205] Tried to move back new op lower bound from 7333791538562822144 to 7333791538493722624. Current Snapshot: MvccSnapshot[applied={T|T < 7333791538562822144}]
I20260927 02:34:09.861996 7901 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:34:09.862156 7901 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:34:09.864292 7901 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e302084e70d14287803dacb09876350e: Sending DeleteTablet for 3 replicas of tablet a9edb14c698546b295595ccb5e23a7cf
I20260927 02:34:09.864995 8319 tablet_service.cc:1514] Processing DeleteTablet for tablet a9edb14c698546b295595ccb5e23a7cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:34:09 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34830
I20260927 02:34:09.865197 8179 tablet_service.cc:1514] Processing DeleteTablet for tablet a9edb14c698546b295595ccb5e23a7cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:34:09 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41258
I20260927 02:34:09.865609 8435 tablet_replica.cc:330] T a9edb14c698546b295595ccb5e23a7cf P cc8fdd96bba24e3682c3ba405fcc0a45: stopping tablet replica
I20260927 02:34:09.865610 8039 tablet_service.cc:1514] Processing DeleteTablet for tablet a9edb14c698546b295595ccb5e23a7cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:34:09 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60972
I20260927 02:34:09.865737 7901 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35068:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 02:34:09.865947 8435 raft_consensus.cc:2229] T a9edb14c698546b295595ccb5e23a7cf P cc8fdd96bba24e3682c3ba405fcc0a45 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260927 02:34:09.866101 7901 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:34:09.866216 8436 tablet_replica.cc:330] T a9edb14c698546b295595ccb5e23a7cf P 619d0542fa744235954689b0f8badebb: stopping tablet replica
I20260927 02:34:09.866245 8435 raft_consensus.cc:2258] T a9edb14c698546b295595ccb5e23a7cf P cc8fdd96bba24e3682c3ba405fcc0a45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:34:09.866462 8436 raft_consensus.cc:2229] T a9edb14c698546b295595ccb5e23a7cf P 619d0542fa744235954689b0f8badebb [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:34:09.866820 8436 raft_consensus.cc:2258] T a9edb14c698546b295595ccb5e23a7cf P 619d0542fa744235954689b0f8badebb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:34:09.867684 8435 ts_tablet_manager.cc:1905] T a9edb14c698546b295595ccb5e23a7cf P cc8fdd96bba24e3682c3ba405fcc0a45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:34:09.868113 8436 ts_tablet_manager.cc:1905] T a9edb14c698546b295595ccb5e23a7cf P 619d0542fa744235954689b0f8badebb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:34:09.870325 8435 ts_tablet_manager.cc:1918] T a9edb14c698546b295595ccb5e23a7cf P cc8fdd96bba24e3682c3ba405fcc0a45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:34:09.870455 8435 log.cc:1196] T a9edb14c698546b295595ccb5e23a7cf P cc8fdd96bba24e3682c3ba405fcc0a45: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-2/wal/wals/a9edb14c698546b295595ccb5e23a7cf
I20260927 02:34:09.870772 8435 ts_tablet_manager.cc:1939] T a9edb14c698546b295595ccb5e23a7cf P cc8fdd96bba24e3682c3ba405fcc0a45: Deleting consensus metadata
I20260927 02:34:09.871551 8039 tablet_service.cc:1467] Processing CreateTablet for tablet e96839830d834262888026518cb10e70 (DEFAULT_TABLE table=test-table [id=bb1a9f9e8e834537a2d12e79573a7d65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:34:09.871845 8039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e96839830d834262888026518cb10e70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:34:09.873212 8179 tablet_service.cc:1467] Processing CreateTablet for tablet e96839830d834262888026518cb10e70 (DEFAULT_TABLE table=test-table [id=bb1a9f9e8e834537a2d12e79573a7d65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:34:09.873430 8179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e96839830d834262888026518cb10e70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:34:09.873845 8414 tablet_bootstrap.cc:492] T e96839830d834262888026518cb10e70 P dfcc410419d7462193cfcc270fc06b54: Bootstrap starting.
I20260927 02:34:09.874141 7888 catalog_manager.cc:4841] TS cc8fdd96bba24e3682c3ba405fcc0a45 (127.0.98.3:45319): tablet a9edb14c698546b295595ccb5e23a7cf (table test-table [id=cecb7c8730fe4490b3f01c458c5073af]) successfully deleted
I20260927 02:34:09.874413 8437 tablet_replica.cc:330] T a9edb14c698546b295595ccb5e23a7cf P dfcc410419d7462193cfcc270fc06b54: stopping tablet replica
I20260927 02:34:09.874703 8437 raft_consensus.cc:2229] T a9edb14c698546b295595ccb5e23a7cf P dfcc410419d7462193cfcc270fc06b54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:34:09.874725 8414 tablet_bootstrap.cc:654] T e96839830d834262888026518cb10e70 P dfcc410419d7462193cfcc270fc06b54: Neither blocks nor log segments found. Creating new log.
I20260927 02:34:09.874855 8319 tablet_service.cc:1467] Processing CreateTablet for tablet e96839830d834262888026518cb10e70 (DEFAULT_TABLE table=test-table [id=bb1a9f9e8e834537a2d12e79573a7d65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:34:09.874982 8437 raft_consensus.cc:2258] T a9edb14c698546b295595ccb5e23a7cf P dfcc410419d7462193cfcc270fc06b54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:34:09.875068 8319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e96839830d834262888026518cb10e70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:34:09.876312 8437 ts_tablet_manager.cc:1905] T a9edb14c698546b295595ccb5e23a7cf P dfcc410419d7462193cfcc270fc06b54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:34:09.877131 8415 tablet_bootstrap.cc:492] T e96839830d834262888026518cb10e70 P cc8fdd96bba24e3682c3ba405fcc0a45: Bootstrap starting.
I20260927 02:34:09.877998 8415 tablet_bootstrap.cc:654] T e96839830d834262888026518cb10e70 P cc8fdd96bba24e3682c3ba405fcc0a45: Neither blocks nor log segments found. Creating new log.
I20260927 02:34:09.878368 8437 ts_tablet_manager.cc:1918] T a9edb14c698546b295595ccb5e23a7cf P dfcc410419d7462193cfcc270fc06b54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:34:09.878427 8437 log.cc:1196] T a9edb14c698546b295595ccb5e23a7cf P dfcc410419d7462193cfcc270fc06b54: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-0/wal/wals/a9edb14c698546b295595ccb5e23a7cf
I20260927 02:34:09.878692 8437 ts_tablet_manager.cc:1939] T a9edb14c698546b295595ccb5e23a7cf P dfcc410419d7462193cfcc270fc06b54: Deleting consensus metadata
I20260927 02:34:09.879156 8415 tablet_bootstrap.cc:492] T e96839830d834262888026518cb10e70 P cc8fdd96bba24e3682c3ba405fcc0a45: No bootstrap required, opened a new log
I20260927 02:34:09.879245 8415 ts_tablet_manager.cc:1397] T e96839830d834262888026518cb10e70 P cc8fdd96bba24e3682c3ba405fcc0a45: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:34:09.879632 8415 raft_consensus.cc:348] T e96839830d834262888026518cb10e70 P cc8fdd96bba24e3682c3ba405fcc0a45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } } peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } }
I20260927 02:34:09.879726 8415 raft_consensus.cc:374] T e96839830d834262888026518cb10e70 P cc8fdd96bba24e3682c3ba405fcc0a45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:34:09.879755 8415 raft_consensus.cc:729] T e96839830d834262888026518cb10e70 P cc8fdd96bba24e3682c3ba405fcc0a45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8fdd96bba24e3682c3ba405fcc0a45, State: Initialized, Role: FOLLOWER
I20260927 02:34:09.879861 8415 consensus_queue.cc:260] T e96839830d834262888026518cb10e70 P cc8fdd96bba24e3682c3ba405fcc0a45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } } peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } }
I20260927 02:34:09.879966 7886 catalog_manager.cc:4841] TS dfcc410419d7462193cfcc270fc06b54 (127.0.98.1:35293): tablet a9edb14c698546b295595ccb5e23a7cf (table test-table [id=cecb7c8730fe4490b3f01c458c5073af]) successfully deleted
I20260927 02:34:09.880086 8415 ts_tablet_manager.cc:1428] T e96839830d834262888026518cb10e70 P cc8fdd96bba24e3682c3ba405fcc0a45: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 02:34:09.880677 8414 tablet_bootstrap.cc:492] T e96839830d834262888026518cb10e70 P dfcc410419d7462193cfcc270fc06b54: No bootstrap required, opened a new log
I20260927 02:34:09.880754 8414 ts_tablet_manager.cc:1397] T e96839830d834262888026518cb10e70 P dfcc410419d7462193cfcc270fc06b54: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260927 02:34:09.881356 8414 raft_consensus.cc:348] T e96839830d834262888026518cb10e70 P dfcc410419d7462193cfcc270fc06b54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } } peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } }
I20260927 02:34:09.881456 8414 raft_consensus.cc:374] T e96839830d834262888026518cb10e70 P dfcc410419d7462193cfcc270fc06b54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:34:09.881485 8414 raft_consensus.cc:729] T e96839830d834262888026518cb10e70 P dfcc410419d7462193cfcc270fc06b54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfcc410419d7462193cfcc270fc06b54, State: Initialized, Role: FOLLOWER
I20260927 02:34:09.881588 8414 consensus_queue.cc:260] T e96839830d834262888026518cb10e70 P dfcc410419d7462193cfcc270fc06b54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } } peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } }
I20260927 02:34:09.881831 8414 ts_tablet_manager.cc:1428] T e96839830d834262888026518cb10e70 P dfcc410419d7462193cfcc270fc06b54: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 02:34:09.886798 8413 tablet_bootstrap.cc:492] T e96839830d834262888026518cb10e70 P 619d0542fa744235954689b0f8badebb: Bootstrap starting.
I20260927 02:34:09.887378 8436 ts_tablet_manager.cc:1918] T a9edb14c698546b295595ccb5e23a7cf P 619d0542fa744235954689b0f8badebb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:34:09.887470 8436 log.cc:1196] T a9edb14c698546b295595ccb5e23a7cf P 619d0542fa744235954689b0f8badebb: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-1/wal/wals/a9edb14c698546b295595ccb5e23a7cf
I20260927 02:34:09.887749 8436 ts_tablet_manager.cc:1939] T a9edb14c698546b295595ccb5e23a7cf P 619d0542fa744235954689b0f8badebb: Deleting consensus metadata
I20260927 02:34:09.887806 8413 tablet_bootstrap.cc:654] T e96839830d834262888026518cb10e70 P 619d0542fa744235954689b0f8badebb: Neither blocks nor log segments found. Creating new log.
I20260927 02:34:09.888870 7889 catalog_manager.cc:4841] TS 619d0542fa744235954689b0f8badebb (127.0.98.2:43319): tablet a9edb14c698546b295595ccb5e23a7cf (table test-table [id=cecb7c8730fe4490b3f01c458c5073af]) successfully deleted
I20260927 02:34:09.889837 8413 tablet_bootstrap.cc:492] T e96839830d834262888026518cb10e70 P 619d0542fa744235954689b0f8badebb: No bootstrap required, opened a new log
I20260927 02:34:09.889919 8413 ts_tablet_manager.cc:1397] T e96839830d834262888026518cb10e70 P 619d0542fa744235954689b0f8badebb: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260927 02:34:09.890304 8413 raft_consensus.cc:348] T e96839830d834262888026518cb10e70 P 619d0542fa744235954689b0f8badebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } } peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } }
I20260927 02:34:09.890398 8413 raft_consensus.cc:374] T e96839830d834262888026518cb10e70 P 619d0542fa744235954689b0f8badebb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:34:09.890419 8413 raft_consensus.cc:729] T e96839830d834262888026518cb10e70 P 619d0542fa744235954689b0f8badebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 619d0542fa744235954689b0f8badebb, State: Initialized, Role: FOLLOWER
I20260927 02:34:09.890529 8413 consensus_queue.cc:260] T e96839830d834262888026518cb10e70 P 619d0542fa744235954689b0f8badebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } } peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } }
I20260927 02:34:09.890798 8413 ts_tablet_manager.cc:1428] T e96839830d834262888026518cb10e70 P 619d0542fa744235954689b0f8badebb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20260927 02:34:09.900519 8106 tablet.cc:2369] T e96839830d834262888026518cb10e70 P dfcc410419d7462193cfcc270fc06b54: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:34:09.913865 8419 raft_consensus.cc:482] T e96839830d834262888026518cb10e70 P 619d0542fa744235954689b0f8badebb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:34:09.913980 8419 raft_consensus.cc:504] T e96839830d834262888026518cb10e70 P 619d0542fa744235954689b0f8badebb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } } peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } }
I20260927 02:34:09.914366 8419 leader_election.cc:288] T e96839830d834262888026518cb10e70 P 619d0542fa744235954689b0f8badebb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfcc410419d7462193cfcc270fc06b54 (127.0.98.1:35293), cc8fdd96bba24e3682c3ba405fcc0a45 (127.0.98.3:45319)
I20260927 02:34:09.914757 8339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e96839830d834262888026518cb10e70" candidate_uuid: "619d0542fa744235954689b0f8badebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" is_pre_election: true
I20260927 02:34:09.914755 8059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e96839830d834262888026518cb10e70" candidate_uuid: "619d0542fa744235954689b0f8badebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfcc410419d7462193cfcc270fc06b54" is_pre_election: true
I20260927 02:34:09.914892 8339 raft_consensus.cc:2454] T e96839830d834262888026518cb10e70 P cc8fdd96bba24e3682c3ba405fcc0a45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 619d0542fa744235954689b0f8badebb in term 0.
I20260927 02:34:09.914911 8059 raft_consensus.cc:2454] T e96839830d834262888026518cb10e70 P dfcc410419d7462193cfcc270fc06b54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 619d0542fa744235954689b0f8badebb in term 0.
I20260927 02:34:09.915194 8130 leader_election.cc:302] T e96839830d834262888026518cb10e70 P 619d0542fa744235954689b0f8badebb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 619d0542fa744235954689b0f8badebb, dfcc410419d7462193cfcc270fc06b54; no voters:
I20260927 02:34:09.915359 8419 raft_consensus.cc:2781] T e96839830d834262888026518cb10e70 P 619d0542fa744235954689b0f8badebb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:34:09.915421 8419 raft_consensus.cc:482] T e96839830d834262888026518cb10e70 P 619d0542fa744235954689b0f8badebb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:34:09.915465 8419 raft_consensus.cc:3037] T e96839830d834262888026518cb10e70 P 619d0542fa744235954689b0f8badebb [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:34:09.916400 8419 raft_consensus.cc:504] T e96839830d834262888026518cb10e70 P 619d0542fa744235954689b0f8badebb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } } peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } }
I20260927 02:34:09.916712 8419 leader_election.cc:288] T e96839830d834262888026518cb10e70 P 619d0542fa744235954689b0f8badebb [CANDIDATE]: Term 1 election: Requested vote from peers dfcc410419d7462193cfcc270fc06b54 (127.0.98.1:35293), cc8fdd96bba24e3682c3ba405fcc0a45 (127.0.98.3:45319)
I20260927 02:34:09.917042 8059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e96839830d834262888026518cb10e70" candidate_uuid: "619d0542fa744235954689b0f8badebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfcc410419d7462193cfcc270fc06b54"
I20260927 02:34:09.917119 8339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e96839830d834262888026518cb10e70" candidate_uuid: "619d0542fa744235954689b0f8badebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45"
I20260927 02:34:09.917172 8059 raft_consensus.cc:3037] T e96839830d834262888026518cb10e70 P dfcc410419d7462193cfcc270fc06b54 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:34:09.917229 8339 raft_consensus.cc:3037] T e96839830d834262888026518cb10e70 P cc8fdd96bba24e3682c3ba405fcc0a45 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:34:09.918188 8059 raft_consensus.cc:2454] T e96839830d834262888026518cb10e70 P dfcc410419d7462193cfcc270fc06b54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 619d0542fa744235954689b0f8badebb in term 1.
I20260927 02:34:09.918188 8339 raft_consensus.cc:2454] T e96839830d834262888026518cb10e70 P cc8fdd96bba24e3682c3ba405fcc0a45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 619d0542fa744235954689b0f8badebb in term 1.
I20260927 02:34:09.918467 8130 leader_election.cc:302] T e96839830d834262888026518cb10e70 P 619d0542fa744235954689b0f8badebb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 619d0542fa744235954689b0f8badebb, dfcc410419d7462193cfcc270fc06b54; no voters:
I20260927 02:34:09.918680 8419 raft_consensus.cc:2781] T e96839830d834262888026518cb10e70 P 619d0542fa744235954689b0f8badebb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:34:09.918766 8419 raft_consensus.cc:686] T e96839830d834262888026518cb10e70 P 619d0542fa744235954689b0f8badebb [term 1 LEADER]: Becoming Leader. State: Replica: 619d0542fa744235954689b0f8badebb, State: Running, Role: LEADER
I20260927 02:34:09.918889 8419 consensus_queue.cc:237] T e96839830d834262888026518cb10e70 P 619d0542fa744235954689b0f8badebb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } } peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } }
I20260927 02:34:09.920296 7900 catalog_manager.cc:5495] T e96839830d834262888026518cb10e70 P 619d0542fa744235954689b0f8badebb reported cstate change: term changed from 0 to 1, leader changed from <none> to 619d0542fa744235954689b0f8badebb (127.0.98.2). New cstate: current_term: 1 leader_uuid: "619d0542fa744235954689b0f8badebb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } health_report { overall_health: UNKNOWN } } }
Sep 27 02:34:09 dist-test-slave-grlk krb5kdc[7857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476448, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:34:09.962878 8447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:09.954573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33027 (local address 127.0.0.1:35072)
0927 02:34:09.954844 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:09.954859 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:09.954958 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:09.955211 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:09.955213 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:09.955570 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:09.955660 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:09.955666 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.956676 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.956679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:09.957289 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:09.957298 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.957838 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.957841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:09.957901 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:09.958430 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:09.958443 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:09.959753 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:34:09.961494 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:09.961501 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:09.961506 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:09.961719 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:09.961986 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:09.961988 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:09.961989 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:09.962055 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:09.962451 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:34:09.962456 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:34:09.962560 (+ 104us) client_negotiation.cc:769] Sending connection context
0927 02:34:09.962647 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 02:34:09.962746 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":94,"threads_started":1}
I20260927 02:34:09.963006 7951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:09.954658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35072 (local address 127.0.98.62:33027)
0927 02:34:09.954800 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:09.954804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:09.954933 (+ 129us) server_negotiation.cc:407] Connection header received
0927 02:34:09.955044 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:09.955047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:09.955095 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:09.955193 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:09.955783 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.956575 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.957406 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:09.957739 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:09.957838 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:09.959883 (+ 2045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:09.959887 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:09.959890 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:09.959914 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:09.961392 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:09.961802 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:09.961806 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:09.961808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:09.961896 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:09.962125 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:09.962128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:09.962129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:09.962341 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:09.962438 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:09.962726 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 02:34:09.962832 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 02:34:09.966626 7900 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35072:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 02:34:09.967427 7900 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:34:09.972007 8039 tablet_service.cc:1467] Processing CreateTablet for tablet 1768010a0fab4ddc99f2ba8a54521f59 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4f1be56162164a2d865a67a048401086]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:34:09.972204 8319 tablet_service.cc:1467] Processing CreateTablet for tablet 1768010a0fab4ddc99f2ba8a54521f59 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4f1be56162164a2d865a67a048401086]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:34:09.972389 8039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1768010a0fab4ddc99f2ba8a54521f59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:34:09.972256 8179 tablet_service.cc:1467] Processing CreateTablet for tablet 1768010a0fab4ddc99f2ba8a54521f59 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4f1be56162164a2d865a67a048401086]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:34:09.972504 8319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1768010a0fab4ddc99f2ba8a54521f59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:34:09.972640 8179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1768010a0fab4ddc99f2ba8a54521f59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:34:09.975221 8414 tablet_bootstrap.cc:492] T 1768010a0fab4ddc99f2ba8a54521f59 P dfcc410419d7462193cfcc270fc06b54: Bootstrap starting.
I20260927 02:34:09.975437 8413 tablet_bootstrap.cc:492] T 1768010a0fab4ddc99f2ba8a54521f59 P 619d0542fa744235954689b0f8badebb: Bootstrap starting.
I20260927 02:34:09.976171 8414 tablet_bootstrap.cc:654] T 1768010a0fab4ddc99f2ba8a54521f59 P dfcc410419d7462193cfcc270fc06b54: Neither blocks nor log segments found. Creating new log.
I20260927 02:34:09.976313 8413 tablet_bootstrap.cc:654] T 1768010a0fab4ddc99f2ba8a54521f59 P 619d0542fa744235954689b0f8badebb: Neither blocks nor log segments found. Creating new log.
I20260927 02:34:09.977011 8415 tablet_bootstrap.cc:492] T 1768010a0fab4ddc99f2ba8a54521f59 P cc8fdd96bba24e3682c3ba405fcc0a45: Bootstrap starting.
I20260927 02:34:09.977509 8414 tablet_bootstrap.cc:492] T 1768010a0fab4ddc99f2ba8a54521f59 P dfcc410419d7462193cfcc270fc06b54: No bootstrap required, opened a new log
I20260927 02:34:09.977598 8414 ts_tablet_manager.cc:1397] T 1768010a0fab4ddc99f2ba8a54521f59 P dfcc410419d7462193cfcc270fc06b54: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:34:09.977875 8415 tablet_bootstrap.cc:654] T 1768010a0fab4ddc99f2ba8a54521f59 P cc8fdd96bba24e3682c3ba405fcc0a45: Neither blocks nor log segments found. Creating new log.
I20260927 02:34:09.978048 8414 raft_consensus.cc:348] T 1768010a0fab4ddc99f2ba8a54521f59 P dfcc410419d7462193cfcc270fc06b54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } } peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } }
I20260927 02:34:09.978173 8414 raft_consensus.cc:374] T 1768010a0fab4ddc99f2ba8a54521f59 P dfcc410419d7462193cfcc270fc06b54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:34:09.978214 8414 raft_consensus.cc:729] T 1768010a0fab4ddc99f2ba8a54521f59 P dfcc410419d7462193cfcc270fc06b54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfcc410419d7462193cfcc270fc06b54, State: Initialized, Role: FOLLOWER
I20260927 02:34:09.978334 8414 consensus_queue.cc:260] T 1768010a0fab4ddc99f2ba8a54521f59 P dfcc410419d7462193cfcc270fc06b54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } } peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } }
I20260927 02:34:09.978614 8414 ts_tablet_manager.cc:1428] T 1768010a0fab4ddc99f2ba8a54521f59 P dfcc410419d7462193cfcc270fc06b54: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:34:09.978904 8420 tablet_replica.cc:439] T 1768010a0fab4ddc99f2ba8a54521f59 P dfcc410419d7462193cfcc270fc06b54: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } } peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } } }
I20260927 02:34:09.978997 8415 tablet_bootstrap.cc:492] T 1768010a0fab4ddc99f2ba8a54521f59 P cc8fdd96bba24e3682c3ba405fcc0a45: No bootstrap required, opened a new log
I20260927 02:34:09.979043 8420 tablet_replica.cc:442] T 1768010a0fab4ddc99f2ba8a54521f59 P dfcc410419d7462193cfcc270fc06b54: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:34:09.979065 8415 ts_tablet_manager.cc:1397] T 1768010a0fab4ddc99f2ba8a54521f59 P cc8fdd96bba24e3682c3ba405fcc0a45: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:34:09.979640 8415 raft_consensus.cc:348] T 1768010a0fab4ddc99f2ba8a54521f59 P cc8fdd96bba24e3682c3ba405fcc0a45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } } peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } }
I20260927 02:34:09.979753 8415 raft_consensus.cc:374] T 1768010a0fab4ddc99f2ba8a54521f59 P cc8fdd96bba24e3682c3ba405fcc0a45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:34:09.979784 8415 raft_consensus.cc:729] T 1768010a0fab4ddc99f2ba8a54521f59 P cc8fdd96bba24e3682c3ba405fcc0a45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8fdd96bba24e3682c3ba405fcc0a45, State: Initialized, Role: FOLLOWER
I20260927 02:34:09.979894 8415 consensus_queue.cc:260] T 1768010a0fab4ddc99f2ba8a54521f59 P cc8fdd96bba24e3682c3ba405fcc0a45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } } peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } }
I20260927 02:34:09.980149 8415 ts_tablet_manager.cc:1428] T 1768010a0fab4ddc99f2ba8a54521f59 P cc8fdd96bba24e3682c3ba405fcc0a45: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:34:09.980165 8421 tablet_replica.cc:439] T 1768010a0fab4ddc99f2ba8a54521f59 P cc8fdd96bba24e3682c3ba405fcc0a45: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } } peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } } }
I20260927 02:34:09.980335 8421 tablet_replica.cc:442] T 1768010a0fab4ddc99f2ba8a54521f59 P cc8fdd96bba24e3682c3ba405fcc0a45: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:34:09.980894 8413 tablet_bootstrap.cc:492] T 1768010a0fab4ddc99f2ba8a54521f59 P 619d0542fa744235954689b0f8badebb: No bootstrap required, opened a new log
I20260927 02:34:09.981000 8413 ts_tablet_manager.cc:1397] T 1768010a0fab4ddc99f2ba8a54521f59 P 619d0542fa744235954689b0f8badebb: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20260927 02:34:09.981357 8413 raft_consensus.cc:348] T 1768010a0fab4ddc99f2ba8a54521f59 P 619d0542fa744235954689b0f8badebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } } peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } }
I20260927 02:34:09.981446 8413 raft_consensus.cc:374] T 1768010a0fab4ddc99f2ba8a54521f59 P 619d0542fa744235954689b0f8badebb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:34:09.981474 8413 raft_consensus.cc:729] T 1768010a0fab4ddc99f2ba8a54521f59 P 619d0542fa744235954689b0f8badebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 619d0542fa744235954689b0f8badebb, State: Initialized, Role: FOLLOWER
I20260927 02:34:09.981594 8413 consensus_queue.cc:260] T 1768010a0fab4ddc99f2ba8a54521f59 P 619d0542fa744235954689b0f8badebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } } peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } }
I20260927 02:34:09.981824 8413 ts_tablet_manager.cc:1428] T 1768010a0fab4ddc99f2ba8a54521f59 P 619d0542fa744235954689b0f8badebb: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 02:34:09.981982 8419 tablet_replica.cc:439] T 1768010a0fab4ddc99f2ba8a54521f59 P 619d0542fa744235954689b0f8badebb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } } peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } } }
I20260927 02:34:09.982074 8419 tablet_replica.cc:442] T 1768010a0fab4ddc99f2ba8a54521f59 P 619d0542fa744235954689b0f8badebb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:34:10.112488 8421 raft_consensus.cc:482] T 1768010a0fab4ddc99f2ba8a54521f59 P cc8fdd96bba24e3682c3ba405fcc0a45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:34:10.112632 8421 raft_consensus.cc:504] T 1768010a0fab4ddc99f2ba8a54521f59 P cc8fdd96bba24e3682c3ba405fcc0a45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } } peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } }
I20260927 02:34:10.114065 8421 leader_election.cc:288] T 1768010a0fab4ddc99f2ba8a54521f59 P cc8fdd96bba24e3682c3ba405fcc0a45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfcc410419d7462193cfcc270fc06b54 (127.0.98.1:35293), 619d0542fa744235954689b0f8badebb (127.0.98.2:43319)
I20260927 02:34:10.119992 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:10.114582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37539 (local address 127.0.98.2:43319)
0927 02:34:10.114740 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:10.114744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:10.115433 (+ 689us) server_negotiation.cc:407] Connection header received
0927 02:34:10.115676 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:10.115679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:10.115730 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:10.115778 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:34:10.116156 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:10.117067 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:10.118668 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:10.119445 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:10.119562 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:10.119636 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:10.119815 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 02:34:10.119885 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 02:34:10.120007 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:10.114316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43319 (local address 127.0.98.3:37539)
0927 02:34:10.115345 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:10.115358 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:10.115530 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:10.115910 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:10.115913 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:10.115929 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:34:10.116034 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:10.116039 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:10.117162 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:10.117166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:10.118478 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:10.118491 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:10.119568 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:10.119571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:10.119680 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:10.119709 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 02:34:10.119785 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 02:34:10.119854 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":959,"mutex_wait_us":64,"spinlock_wait_cycles":768,"thread_start_us":45,"threads_started":1}
I20260927 02:34:10.120739 8199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1768010a0fab4ddc99f2ba8a54521f59" candidate_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "619d0542fa744235954689b0f8badebb" is_pre_election: true
I20260927 02:34:10.121033 8199 raft_consensus.cc:2454] T 1768010a0fab4ddc99f2ba8a54521f59 P 619d0542fa744235954689b0f8badebb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc8fdd96bba24e3682c3ba405fcc0a45 in term 0.
I20260927 02:34:10.121476 8452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:10.114279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35293 (local address 127.0.98.3:34921)
0927 02:34:10.114663 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:10.114679 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:10.114783 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:10.115450 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:10.115453 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:10.115473 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:34:10.115616 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:10.115623 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:10.117094 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:10.117098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:10.118125 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:10.118136 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:10.121017 (+ 2881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:10.121021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:10.121138 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:10.121175 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 02:34:10.121248 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 02:34:10.121322 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":1,"spinlock_wait_cycles":29952,"thread_start_us":91,"threads_started":1}
I20260927 02:34:10.121577 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:10.115133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34921 (local address 127.0.98.1:35293)
0927 02:34:10.115248 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:10.115251 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:10.115264 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:34:10.115309 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:10.115311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:10.115348 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:10.115414 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:34:10.115739 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:10.116966 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:10.118271 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:10.120750 (+ 2479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:10.120836 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:10.120897 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:10.121378 (+ 481us) server_negotiation.cc:299] Negotiation successful
0927 02:34:10.121453 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 02:34:10.121464 8273 leader_election.cc:302] T 1768010a0fab4ddc99f2ba8a54521f59 P cc8fdd96bba24e3682c3ba405fcc0a45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 619d0542fa744235954689b0f8badebb, cc8fdd96bba24e3682c3ba405fcc0a45; no voters:
I20260927 02:34:10.121819 8421 raft_consensus.cc:2781] T 1768010a0fab4ddc99f2ba8a54521f59 P cc8fdd96bba24e3682c3ba405fcc0a45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:34:10.121882 8421 raft_consensus.cc:482] T 1768010a0fab4ddc99f2ba8a54521f59 P cc8fdd96bba24e3682c3ba405fcc0a45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:34:10.121927 8421 raft_consensus.cc:3037] T 1768010a0fab4ddc99f2ba8a54521f59 P cc8fdd96bba24e3682c3ba405fcc0a45 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:34:10.123159 8421 raft_consensus.cc:504] T 1768010a0fab4ddc99f2ba8a54521f59 P cc8fdd96bba24e3682c3ba405fcc0a45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } } peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } }
I20260927 02:34:10.123555 8421 leader_election.cc:288] T 1768010a0fab4ddc99f2ba8a54521f59 P cc8fdd96bba24e3682c3ba405fcc0a45 [CANDIDATE]: Term 1 election: Requested vote from peers dfcc410419d7462193cfcc270fc06b54 (127.0.98.1:35293), 619d0542fa744235954689b0f8badebb (127.0.98.2:43319)
I20260927 02:34:10.124166 8059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1768010a0fab4ddc99f2ba8a54521f59" candidate_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfcc410419d7462193cfcc270fc06b54" is_pre_election: true
I20260927 02:34:10.124260 8199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1768010a0fab4ddc99f2ba8a54521f59" candidate_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "619d0542fa744235954689b0f8badebb"
I20260927 02:34:10.124311 8059 raft_consensus.cc:2454] T 1768010a0fab4ddc99f2ba8a54521f59 P dfcc410419d7462193cfcc270fc06b54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc8fdd96bba24e3682c3ba405fcc0a45 in term 0.
I20260927 02:34:10.124359 8199 raft_consensus.cc:3037] T 1768010a0fab4ddc99f2ba8a54521f59 P 619d0542fa744235954689b0f8badebb [term 0 FOLLOWER]: Advancing to term 1
W20260927 02:34:10.125013 8246 tablet.cc:2369] T 1768010a0fab4ddc99f2ba8a54521f59 P 619d0542fa744235954689b0f8badebb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:34:10.125520 8199 raft_consensus.cc:2454] T 1768010a0fab4ddc99f2ba8a54521f59 P 619d0542fa744235954689b0f8badebb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc8fdd96bba24e3682c3ba405fcc0a45 in term 1.
I20260927 02:34:10.124178 8058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1768010a0fab4ddc99f2ba8a54521f59" candidate_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfcc410419d7462193cfcc270fc06b54"
I20260927 02:34:10.125773 8058 raft_consensus.cc:3037] T 1768010a0fab4ddc99f2ba8a54521f59 P dfcc410419d7462193cfcc270fc06b54 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:34:10.126901 8058 raft_consensus.cc:2454] T 1768010a0fab4ddc99f2ba8a54521f59 P dfcc410419d7462193cfcc270fc06b54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc8fdd96bba24e3682c3ba405fcc0a45 in term 1.
I20260927 02:34:10.127296 8273 leader_election.cc:302] T 1768010a0fab4ddc99f2ba8a54521f59 P cc8fdd96bba24e3682c3ba405fcc0a45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 619d0542fa744235954689b0f8badebb, cc8fdd96bba24e3682c3ba405fcc0a45; no voters:
I20260927 02:34:10.127444 8421 raft_consensus.cc:2781] T 1768010a0fab4ddc99f2ba8a54521f59 P cc8fdd96bba24e3682c3ba405fcc0a45 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:34:10.127607 8421 raft_consensus.cc:686] T 1768010a0fab4ddc99f2ba8a54521f59 P cc8fdd96bba24e3682c3ba405fcc0a45 [term 1 LEADER]: Becoming Leader. State: Replica: cc8fdd96bba24e3682c3ba405fcc0a45, State: Running, Role: LEADER
I20260927 02:34:10.127840 8421 consensus_queue.cc:237] T 1768010a0fab4ddc99f2ba8a54521f59 P cc8fdd96bba24e3682c3ba405fcc0a45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } } peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } }
I20260927 02:34:10.129789 8453 tablet_replica.cc:439] T 1768010a0fab4ddc99f2ba8a54521f59 P cc8fdd96bba24e3682c3ba405fcc0a45: TxnStatusTablet state changed. Reason: New leader cc8fdd96bba24e3682c3ba405fcc0a45. Latest consensus state: current_term: 1 leader_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } } peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } } }
I20260927 02:34:10.129881 8453 tablet_replica.cc:442] T 1768010a0fab4ddc99f2ba8a54521f59 P cc8fdd96bba24e3682c3ba405fcc0a45: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:34:10.130123 8455 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 02:34:10.130110 7900 catalog_manager.cc:5495] T 1768010a0fab4ddc99f2ba8a54521f59 P cc8fdd96bba24e3682c3ba405fcc0a45 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc8fdd96bba24e3682c3ba405fcc0a45 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } health_report { overall_health: HEALTHY } } }
I20260927 02:34:10.130275 8455 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 02:34:10.174468 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:10.171137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34844 (local address 127.0.98.3:45319)
0927 02:34:10.171266 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:10.171271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:10.171285 (+ 14us) server_negotiation.cc:407] Connection header received
0927 02:34:10.171409 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:10.171411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:10.171462 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:10.171541 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:34:10.171768 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:10.172259 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:10.173128 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:10.173344 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:10.173429 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:10.173697 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:34:10.174104 (+ 407us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:34:10.174199 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:10.174279 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 02:34:10.174330 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 02:34:10.174468 8447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:10.170988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45319 (local address 127.0.0.1:34844)
0927 02:34:10.171152 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:10.171165 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:10.171267 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:10.171565 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:10.171568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:10.171577 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:34:10.171675 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:10.171680 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:10.172357 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:10.172360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:10.173017 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:10.173026 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:10.173437 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:10.173440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:10.173523 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:10.173567 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:34:10.174195 (+ 628us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:34:10.174208 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 02:34:10.174269 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 02:34:10.174352 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 02:34:10.177106 8058 raft_consensus.cc:1261] T 1768010a0fab4ddc99f2ba8a54521f59 P dfcc410419d7462193cfcc270fc06b54 [term 1 FOLLOWER]: Refusing update from remote peer cc8fdd96bba24e3682c3ba405fcc0a45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:34:10.177333 8199 raft_consensus.cc:1261] T 1768010a0fab4ddc99f2ba8a54521f59 P 619d0542fa744235954689b0f8badebb [term 1 FOLLOWER]: Refusing update from remote peer cc8fdd96bba24e3682c3ba405fcc0a45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:34:10.177870 8421 consensus_queue.cc:1035] T 1768010a0fab4ddc99f2ba8a54521f59 P cc8fdd96bba24e3682c3ba405fcc0a45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:34:10.178069 8453 consensus_queue.cc:1035] T 1768010a0fab4ddc99f2ba8a54521f59 P cc8fdd96bba24e3682c3ba405fcc0a45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:34:10.179592 8419 tablet_replica.cc:439] T 1768010a0fab4ddc99f2ba8a54521f59 P 619d0542fa744235954689b0f8badebb: TxnStatusTablet state changed. Reason: New leader cc8fdd96bba24e3682c3ba405fcc0a45. Latest consensus state: current_term: 1 leader_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } } peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } } }
I20260927 02:34:10.179694 8419 tablet_replica.cc:442] T 1768010a0fab4ddc99f2ba8a54521f59 P 619d0542fa744235954689b0f8badebb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:34:10.179662 8420 tablet_replica.cc:439] T 1768010a0fab4ddc99f2ba8a54521f59 P dfcc410419d7462193cfcc270fc06b54: TxnStatusTablet state changed. Reason: New leader cc8fdd96bba24e3682c3ba405fcc0a45. Latest consensus state: current_term: 1 leader_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } } peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } } }
I20260927 02:34:10.179769 8420 tablet_replica.cc:442] T 1768010a0fab4ddc99f2ba8a54521f59 P dfcc410419d7462193cfcc270fc06b54: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:34:10.181866 8453 tablet_replica.cc:439] T 1768010a0fab4ddc99f2ba8a54521f59 P cc8fdd96bba24e3682c3ba405fcc0a45: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } } peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } } }
I20260927 02:34:10.181970 8453 tablet_replica.cc:442] T 1768010a0fab4ddc99f2ba8a54521f59 P cc8fdd96bba24e3682c3ba405fcc0a45: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:34:10.181934 8421 tablet_replica.cc:439] T 1768010a0fab4ddc99f2ba8a54521f59 P cc8fdd96bba24e3682c3ba405fcc0a45: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } } peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } } }
I20260927 02:34:10.182030 8421 tablet_replica.cc:442] T 1768010a0fab4ddc99f2ba8a54521f59 P cc8fdd96bba24e3682c3ba405fcc0a45: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:34:10.182456 8428 mvcc.cc:205] Tried to move back new op lower bound from 7333791539920564224 to 7333791539726290944. Current Snapshot: MvccSnapshot[applied={T|T < 7333791539920564224}]
I20260927 02:34:10.182633 8420 tablet_replica.cc:439] T 1768010a0fab4ddc99f2ba8a54521f59 P dfcc410419d7462193cfcc270fc06b54: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } } peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } } }
I20260927 02:34:10.182721 8420 tablet_replica.cc:442] T 1768010a0fab4ddc99f2ba8a54521f59 P dfcc410419d7462193cfcc270fc06b54: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:34:10.185252 8426 mvcc.cc:205] Tried to move back new op lower bound from 7333791539920564224 to 7333791539726290944. Current Snapshot: MvccSnapshot[applied={T|T < 7333791539920564224}]
I20260927 02:34:10.185390 8419 tablet_replica.cc:439] T 1768010a0fab4ddc99f2ba8a54521f59 P 619d0542fa744235954689b0f8badebb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } } peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } } }
I20260927 02:34:10.185469 8419 tablet_replica.cc:442] T 1768010a0fab4ddc99f2ba8a54521f59 P 619d0542fa744235954689b0f8badebb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:34:10.196365 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:10.192867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41073 (local address 127.0.98.3:45319)
0927 02:34:10.192994 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:10.192997 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:10.193280 (+ 283us) server_negotiation.cc:407] Connection header received
0927 02:34:10.193422 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:10.193424 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:10.193464 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:10.193523 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:34:10.193786 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:10.194323 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:10.195479 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:10.195683 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:10.195754 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:10.196018 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:34:10.196077 (+ 59us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:34:10.196128 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:10.196226 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 02:34:10.196264 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 02:34:10.196441 8463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:10.192803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45319 (local address 127.0.98.2:41073)
0927 02:34:10.193207 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:10.193221 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:10.193316 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:10.193562 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:10.193565 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:10.193572 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:34:10.193678 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:10.193684 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:10.194405 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:10.194408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:10.195379 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:10.195389 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:10.195778 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:10.195782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:10.195866 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:10.195913 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:34:10.196143 (+ 230us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:34:10.196148 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:34:10.196219 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 02:34:10.196299 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":127,"threads_started":1}
I20260927 02:34:10.202307 8339 raft_consensus.cc:1261] T e96839830d834262888026518cb10e70 P cc8fdd96bba24e3682c3ba405fcc0a45 [term 1 FOLLOWER]: Refusing update from remote peer 619d0542fa744235954689b0f8badebb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:34:10.202664 8423 consensus_queue.cc:1035] T e96839830d834262888026518cb10e70 P 619d0542fa744235954689b0f8badebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:34:10.202713 8058 raft_consensus.cc:1261] T e96839830d834262888026518cb10e70 P dfcc410419d7462193cfcc270fc06b54 [term 1 FOLLOWER]: Refusing update from remote peer 619d0542fa744235954689b0f8badebb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:34:10.203112 8423 consensus_queue.cc:1035] T e96839830d834262888026518cb10e70 P 619d0542fa744235954689b0f8badebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:34:10.222558 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:10.218882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41335 (local address 127.0.98.2:43319)
0927 02:34:10.218990 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:10.218995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:10.219178 (+ 183us) server_negotiation.cc:407] Connection header received
0927 02:34:10.219285 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:10.219288 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:10.219333 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:10.219393 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:34:10.219744 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:10.220411 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:10.221603 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:10.221813 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:10.221887 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:10.222160 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:34:10.222221 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:34:10.222268 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:10.222362 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 02:34:10.222427 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 02:34:10.222589 8467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:10.218825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43319 (local address 127.0.98.3:41335)
0927 02:34:10.219098 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:10.219109 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:10.219208 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:10.219407 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:10.219410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:10.219417 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:34:10.219660 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:10.219668 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:10.220516 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:10.220519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:10.221501 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:10.221512 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:10.221906 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:10.221910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:10.222005 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:10.222054 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:34:10.222287 (+ 233us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:34:10.222293 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 02:34:10.222357 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 02:34:10.222436 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":96,"threads_started":1}
I20260927 02:34:10.241524 7900 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:34:10.241694 7900 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:34:10.243937 7900 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e302084e70d14287803dacb09876350e: Sending DeleteTablet for 3 replicas of tablet e96839830d834262888026518cb10e70
I20260927 02:34:10.244482 8039 tablet_service.cc:1514] Processing DeleteTablet for tablet e96839830d834262888026518cb10e70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:34:10 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60972
I20260927 02:34:10.244673 8437 tablet_replica.cc:330] T e96839830d834262888026518cb10e70 P dfcc410419d7462193cfcc270fc06b54: stopping tablet replica
I20260927 02:34:10.244783 8437 raft_consensus.cc:2229] T e96839830d834262888026518cb10e70 P dfcc410419d7462193cfcc270fc06b54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:34:10.244879 8437 raft_consensus.cc:2258] T e96839830d834262888026518cb10e70 P dfcc410419d7462193cfcc270fc06b54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:34:10.245147 8319 tablet_service.cc:1514] Processing DeleteTablet for tablet e96839830d834262888026518cb10e70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:34:10 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34830
I20260927 02:34:10.245303 8435 tablet_replica.cc:330] T e96839830d834262888026518cb10e70 P cc8fdd96bba24e3682c3ba405fcc0a45: stopping tablet replica
I20260927 02:34:10.245395 8435 raft_consensus.cc:2229] T e96839830d834262888026518cb10e70 P cc8fdd96bba24e3682c3ba405fcc0a45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:34:10.245497 8435 raft_consensus.cc:2258] T e96839830d834262888026518cb10e70 P cc8fdd96bba24e3682c3ba405fcc0a45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:34:10.247902 8435 ts_tablet_manager.cc:1905] T e96839830d834262888026518cb10e70 P cc8fdd96bba24e3682c3ba405fcc0a45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:34:10.249135 8179 tablet_service.cc:1514] Processing DeleteTablet for tablet e96839830d834262888026518cb10e70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:34:10 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41258
I20260927 02:34:10.249317 8436 tablet_replica.cc:330] T e96839830d834262888026518cb10e70 P 619d0542fa744235954689b0f8badebb: stopping tablet replica
I20260927 02:34:10.249444 8436 raft_consensus.cc:2229] T e96839830d834262888026518cb10e70 P 619d0542fa744235954689b0f8badebb [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:34:10.249630 8436 raft_consensus.cc:2258] T e96839830d834262888026518cb10e70 P 619d0542fa744235954689b0f8badebb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:34:10.250366 8436 ts_tablet_manager.cc:1905] T e96839830d834262888026518cb10e70 P 619d0542fa744235954689b0f8badebb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:34:10.250442 8437 ts_tablet_manager.cc:1905] T e96839830d834262888026518cb10e70 P dfcc410419d7462193cfcc270fc06b54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:34:10.250461 8435 ts_tablet_manager.cc:1918] T e96839830d834262888026518cb10e70 P cc8fdd96bba24e3682c3ba405fcc0a45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:34:10.250520 8435 log.cc:1196] T e96839830d834262888026518cb10e70 P cc8fdd96bba24e3682c3ba405fcc0a45: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-2/wal/wals/e96839830d834262888026518cb10e70
I20260927 02:34:10.250751 8435 ts_tablet_manager.cc:1939] T e96839830d834262888026518cb10e70 P cc8fdd96bba24e3682c3ba405fcc0a45: Deleting consensus metadata
I20260927 02:34:10.252521 8436 ts_tablet_manager.cc:1918] T e96839830d834262888026518cb10e70 P 619d0542fa744235954689b0f8badebb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:34:10.252570 8436 log.cc:1196] T e96839830d834262888026518cb10e70 P 619d0542fa744235954689b0f8badebb: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-1/wal/wals/e96839830d834262888026518cb10e70
I20260927 02:34:10.252722 8436 ts_tablet_manager.cc:1939] T e96839830d834262888026518cb10e70 P 619d0542fa744235954689b0f8badebb: Deleting consensus metadata
I20260927 02:34:10.252996 7888 catalog_manager.cc:4841] TS cc8fdd96bba24e3682c3ba405fcc0a45 (127.0.98.3:45319): tablet e96839830d834262888026518cb10e70 (table test-table [id=bb1a9f9e8e834537a2d12e79573a7d65]) successfully deleted
I20260927 02:34:10.253197 8437 ts_tablet_manager.cc:1918] T e96839830d834262888026518cb10e70 P dfcc410419d7462193cfcc270fc06b54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:34:10.253279 8437 log.cc:1196] T e96839830d834262888026518cb10e70 P dfcc410419d7462193cfcc270fc06b54: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-0/wal/wals/e96839830d834262888026518cb10e70
I20260927 02:34:10.253525 8437 ts_tablet_manager.cc:1939] T e96839830d834262888026518cb10e70 P dfcc410419d7462193cfcc270fc06b54: Deleting consensus metadata
I20260927 02:34:10.253664 7889 catalog_manager.cc:4841] TS 619d0542fa744235954689b0f8badebb (127.0.98.2:43319): tablet e96839830d834262888026518cb10e70 (table test-table [id=bb1a9f9e8e834537a2d12e79573a7d65]) successfully deleted
I20260927 02:34:10.254163 7886 catalog_manager.cc:4841] TS dfcc410419d7462193cfcc270fc06b54 (127.0.98.1:35293): tablet e96839830d834262888026518cb10e70 (table test-table [id=bb1a9f9e8e834537a2d12e79573a7d65]) successfully deleted
Sep 27 02:34:10 dist-test-slave-grlk krb5kdc[7857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 02:34:10 dist-test-slave-grlk krb5kdc[7857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:10 dist-test-slave-grlk krb5kdc[7857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 02:34:10.274950 7951 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:10.268254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35086 (local address 127.0.98.62:33027)
0927 02:34:10.268386 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:10.268391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:10.268565 (+ 174us) server_negotiation.cc:407] Connection header received
0927 02:34:10.268706 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:10.268709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:10.268753 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:10.268819 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:10.269730 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:10.270242 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:10.271472 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:10.271679 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:10.271748 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:10.274825 (+ 3077us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35086: BlockingRecv error: recv got EOF from 127.0.0.1:35086 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 02:34:10 dist-test-slave-grlk krb5kdc[7857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:34:10.283759 7951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:10.276218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35094 (local address 127.0.98.62:33027)
0927 02:34:10.276316 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:10.276320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:10.276522 (+ 202us) server_negotiation.cc:407] Connection header received
0927 02:34:10.276626 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:10.276629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:10.276669 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:10.276724 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:10.277317 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:10.277861 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:10.278616 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:10.278824 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:10.278905 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:10.281062 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:10.281066 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:10.281068 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:10.281091 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:10.282305 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:10.282660 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:10.282664 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:10.282666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:10.282739 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:10.282968 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:10.282971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:10.282973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:10.283136 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:10.283243 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:10.283516 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 02:34:10.283625 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 02:34:10.286123 7900 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35094:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:34:10.286522 7900 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:34:10.290956 8039 tablet_service.cc:1467] Processing CreateTablet for tablet f395b89623614d9990bc06b00ad24021 (DEFAULT_TABLE table=test-table [id=c6456644988240c1ac2a0955b643c30f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:34:10.291069 8179 tablet_service.cc:1467] Processing CreateTablet for tablet f395b89623614d9990bc06b00ad24021 (DEFAULT_TABLE table=test-table [id=c6456644988240c1ac2a0955b643c30f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:34:10.291131 8319 tablet_service.cc:1467] Processing CreateTablet for tablet f395b89623614d9990bc06b00ad24021 (DEFAULT_TABLE table=test-table [id=c6456644988240c1ac2a0955b643c30f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:34:10.291226 8039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f395b89623614d9990bc06b00ad24021. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:34:10.291342 8179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f395b89623614d9990bc06b00ad24021. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:34:10.291342 8319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f395b89623614d9990bc06b00ad24021. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:34:10.293624 8414 tablet_bootstrap.cc:492] T f395b89623614d9990bc06b00ad24021 P dfcc410419d7462193cfcc270fc06b54: Bootstrap starting.
I20260927 02:34:10.293659 8413 tablet_bootstrap.cc:492] T f395b89623614d9990bc06b00ad24021 P 619d0542fa744235954689b0f8badebb: Bootstrap starting.
I20260927 02:34:10.293926 8415 tablet_bootstrap.cc:492] T f395b89623614d9990bc06b00ad24021 P cc8fdd96bba24e3682c3ba405fcc0a45: Bootstrap starting.
I20260927 02:34:10.294461 8413 tablet_bootstrap.cc:654] T f395b89623614d9990bc06b00ad24021 P 619d0542fa744235954689b0f8badebb: Neither blocks nor log segments found. Creating new log.
I20260927 02:34:10.294492 8414 tablet_bootstrap.cc:654] T f395b89623614d9990bc06b00ad24021 P dfcc410419d7462193cfcc270fc06b54: Neither blocks nor log segments found. Creating new log.
I20260927 02:34:10.294625 8415 tablet_bootstrap.cc:654] T f395b89623614d9990bc06b00ad24021 P cc8fdd96bba24e3682c3ba405fcc0a45: Neither blocks nor log segments found. Creating new log.
I20260927 02:34:10.295732 8413 tablet_bootstrap.cc:492] T f395b89623614d9990bc06b00ad24021 P 619d0542fa744235954689b0f8badebb: No bootstrap required, opened a new log
I20260927 02:34:10.295811 8413 ts_tablet_manager.cc:1397] T f395b89623614d9990bc06b00ad24021 P 619d0542fa744235954689b0f8badebb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 02:34:10.296075 8414 tablet_bootstrap.cc:492] T f395b89623614d9990bc06b00ad24021 P dfcc410419d7462193cfcc270fc06b54: No bootstrap required, opened a new log
I20260927 02:34:10.296142 8414 ts_tablet_manager.cc:1397] T f395b89623614d9990bc06b00ad24021 P dfcc410419d7462193cfcc270fc06b54: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 02:34:10.296139 8415 tablet_bootstrap.cc:492] T f395b89623614d9990bc06b00ad24021 P cc8fdd96bba24e3682c3ba405fcc0a45: No bootstrap required, opened a new log
I20260927 02:34:10.296209 8415 ts_tablet_manager.cc:1397] T f395b89623614d9990bc06b00ad24021 P cc8fdd96bba24e3682c3ba405fcc0a45: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:34:10.296178 8413 raft_consensus.cc:348] T f395b89623614d9990bc06b00ad24021 P 619d0542fa744235954689b0f8badebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } } peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } }
I20260927 02:34:10.296272 8413 raft_consensus.cc:374] T f395b89623614d9990bc06b00ad24021 P 619d0542fa744235954689b0f8badebb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:34:10.296303 8413 raft_consensus.cc:729] T f395b89623614d9990bc06b00ad24021 P 619d0542fa744235954689b0f8badebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 619d0542fa744235954689b0f8badebb, State: Initialized, Role: FOLLOWER
I20260927 02:34:10.296411 8413 consensus_queue.cc:260] T f395b89623614d9990bc06b00ad24021 P 619d0542fa744235954689b0f8badebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } } peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } }
I20260927 02:34:10.296566 8415 raft_consensus.cc:348] T f395b89623614d9990bc06b00ad24021 P cc8fdd96bba24e3682c3ba405fcc0a45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } } peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } }
I20260927 02:34:10.296655 8415 raft_consensus.cc:374] T f395b89623614d9990bc06b00ad24021 P cc8fdd96bba24e3682c3ba405fcc0a45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:34:10.296658 8413 ts_tablet_manager.cc:1428] T f395b89623614d9990bc06b00ad24021 P 619d0542fa744235954689b0f8badebb: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 02:34:10.296686 8415 raft_consensus.cc:729] T f395b89623614d9990bc06b00ad24021 P cc8fdd96bba24e3682c3ba405fcc0a45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8fdd96bba24e3682c3ba405fcc0a45, State: Initialized, Role: FOLLOWER
I20260927 02:34:10.296626 8414 raft_consensus.cc:348] T f395b89623614d9990bc06b00ad24021 P dfcc410419d7462193cfcc270fc06b54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } } peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } }
I20260927 02:34:10.296730 8414 raft_consensus.cc:374] T f395b89623614d9990bc06b00ad24021 P dfcc410419d7462193cfcc270fc06b54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:34:10.296756 8414 raft_consensus.cc:729] T f395b89623614d9990bc06b00ad24021 P dfcc410419d7462193cfcc270fc06b54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfcc410419d7462193cfcc270fc06b54, State: Initialized, Role: FOLLOWER
I20260927 02:34:10.296790 8415 consensus_queue.cc:260] T f395b89623614d9990bc06b00ad24021 P cc8fdd96bba24e3682c3ba405fcc0a45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } } peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } }
I20260927 02:34:10.296859 8414 consensus_queue.cc:260] T f395b89623614d9990bc06b00ad24021 P dfcc410419d7462193cfcc270fc06b54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } } peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } }
I20260927 02:34:10.297055 8415 ts_tablet_manager.cc:1428] T f395b89623614d9990bc06b00ad24021 P cc8fdd96bba24e3682c3ba405fcc0a45: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:34:10.297086 8414 ts_tablet_manager.cc:1428] T f395b89623614d9990bc06b00ad24021 P dfcc410419d7462193cfcc270fc06b54: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:34:10.418339 8469 raft_consensus.cc:482] T f395b89623614d9990bc06b00ad24021 P 619d0542fa744235954689b0f8badebb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:34:10.418465 8469 raft_consensus.cc:504] T f395b89623614d9990bc06b00ad24021 P 619d0542fa744235954689b0f8badebb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } } peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } }
I20260927 02:34:10.418844 8469 leader_election.cc:288] T f395b89623614d9990bc06b00ad24021 P 619d0542fa744235954689b0f8badebb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfcc410419d7462193cfcc270fc06b54 (127.0.98.1:35293), cc8fdd96bba24e3682c3ba405fcc0a45 (127.0.98.3:45319)
I20260927 02:34:10.419234 8339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f395b89623614d9990bc06b00ad24021" candidate_uuid: "619d0542fa744235954689b0f8badebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" is_pre_election: true
I20260927 02:34:10.419268 8058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f395b89623614d9990bc06b00ad24021" candidate_uuid: "619d0542fa744235954689b0f8badebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfcc410419d7462193cfcc270fc06b54" is_pre_election: true
I20260927 02:34:10.419363 8339 raft_consensus.cc:2454] T f395b89623614d9990bc06b00ad24021 P cc8fdd96bba24e3682c3ba405fcc0a45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 619d0542fa744235954689b0f8badebb in term 0.
I20260927 02:34:10.419425 8058 raft_consensus.cc:2454] T f395b89623614d9990bc06b00ad24021 P dfcc410419d7462193cfcc270fc06b54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 619d0542fa744235954689b0f8badebb in term 0.
I20260927 02:34:10.419642 8132 leader_election.cc:302] T f395b89623614d9990bc06b00ad24021 P 619d0542fa744235954689b0f8badebb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 619d0542fa744235954689b0f8badebb, cc8fdd96bba24e3682c3ba405fcc0a45; no voters:
I20260927 02:34:10.419809 8469 raft_consensus.cc:2781] T f395b89623614d9990bc06b00ad24021 P 619d0542fa744235954689b0f8badebb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:34:10.419876 8469 raft_consensus.cc:482] T f395b89623614d9990bc06b00ad24021 P 619d0542fa744235954689b0f8badebb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:34:10.419924 8469 raft_consensus.cc:3037] T f395b89623614d9990bc06b00ad24021 P 619d0542fa744235954689b0f8badebb [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:34:10.420857 8469 raft_consensus.cc:504] T f395b89623614d9990bc06b00ad24021 P 619d0542fa744235954689b0f8badebb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } } peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } }
I20260927 02:34:10.421207 8469 leader_election.cc:288] T f395b89623614d9990bc06b00ad24021 P 619d0542fa744235954689b0f8badebb [CANDIDATE]: Term 1 election: Requested vote from peers dfcc410419d7462193cfcc270fc06b54 (127.0.98.1:35293), cc8fdd96bba24e3682c3ba405fcc0a45 (127.0.98.3:45319)
I20260927 02:34:10.421495 8058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f395b89623614d9990bc06b00ad24021" candidate_uuid: "619d0542fa744235954689b0f8badebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfcc410419d7462193cfcc270fc06b54"
I20260927 02:34:10.421553 8339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f395b89623614d9990bc06b00ad24021" candidate_uuid: "619d0542fa744235954689b0f8badebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45"
I20260927 02:34:10.421599 8058 raft_consensus.cc:3037] T f395b89623614d9990bc06b00ad24021 P dfcc410419d7462193cfcc270fc06b54 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:34:10.421656 8339 raft_consensus.cc:3037] T f395b89623614d9990bc06b00ad24021 P cc8fdd96bba24e3682c3ba405fcc0a45 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:34:10.422468 8058 raft_consensus.cc:2454] T f395b89623614d9990bc06b00ad24021 P dfcc410419d7462193cfcc270fc06b54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 619d0542fa744235954689b0f8badebb in term 1.
I20260927 02:34:10.422529 8339 raft_consensus.cc:2454] T f395b89623614d9990bc06b00ad24021 P cc8fdd96bba24e3682c3ba405fcc0a45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 619d0542fa744235954689b0f8badebb in term 1.
I20260927 02:34:10.422719 8130 leader_election.cc:302] T f395b89623614d9990bc06b00ad24021 P 619d0542fa744235954689b0f8badebb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 619d0542fa744235954689b0f8badebb, dfcc410419d7462193cfcc270fc06b54; no voters:
I20260927 02:34:10.422874 8469 raft_consensus.cc:2781] T f395b89623614d9990bc06b00ad24021 P 619d0542fa744235954689b0f8badebb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:34:10.422959 8469 raft_consensus.cc:686] T f395b89623614d9990bc06b00ad24021 P 619d0542fa744235954689b0f8badebb [term 1 LEADER]: Becoming Leader. State: Replica: 619d0542fa744235954689b0f8badebb, State: Running, Role: LEADER
I20260927 02:34:10.423095 8469 consensus_queue.cc:237] T f395b89623614d9990bc06b00ad24021 P 619d0542fa744235954689b0f8badebb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } } peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } }
I20260927 02:34:10.424568 7902 catalog_manager.cc:5495] T f395b89623614d9990bc06b00ad24021 P 619d0542fa744235954689b0f8badebb reported cstate change: term changed from 0 to 1, leader changed from <none> to 619d0542fa744235954689b0f8badebb (127.0.98.2). New cstate: current_term: 1 leader_uuid: "619d0542fa744235954689b0f8badebb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } health_report { overall_health: UNKNOWN } } }
I20260927 02:34:10.453450 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:10.450799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41280 (local address 127.0.98.2:43319)
0927 02:34:10.450908 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:10.450912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:10.450928 (+ 16us) server_negotiation.cc:407] Connection header received
0927 02:34:10.450987 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:10.450989 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:10.451033 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:10.451092 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:34:10.451284 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:10.451765 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:10.452531 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:10.452736 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:10.452808 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:10.453056 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:34:10.453123 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:34:10.453174 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:10.453281 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 02:34:10.453337 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 02:34:10.455355 8339 raft_consensus.cc:1261] T f395b89623614d9990bc06b00ad24021 P cc8fdd96bba24e3682c3ba405fcc0a45 [term 1 FOLLOWER]: Refusing update from remote peer 619d0542fa744235954689b0f8badebb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:34:10.455397 8058 raft_consensus.cc:1261] T f395b89623614d9990bc06b00ad24021 P dfcc410419d7462193cfcc270fc06b54 [term 1 FOLLOWER]: Refusing update from remote peer 619d0542fa744235954689b0f8badebb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:34:10.455825 8423 consensus_queue.cc:1035] T f395b89623614d9990bc06b00ad24021 P 619d0542fa744235954689b0f8badebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:34:10.455961 8469 consensus_queue.cc:1035] T f395b89623614d9990bc06b00ad24021 P 619d0542fa744235954689b0f8badebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:34:10.462646 7902 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:34:10.462773 7902 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:34:10.464402 7902 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e302084e70d14287803dacb09876350e: Sending DeleteTablet for 3 replicas of tablet f395b89623614d9990bc06b00ad24021
I20260927 02:34:10.464864 8039 tablet_service.cc:1514] Processing DeleteTablet for tablet f395b89623614d9990bc06b00ad24021 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:34:10 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60972
I20260927 02:34:10.464910 8319 tablet_service.cc:1514] Processing DeleteTablet for tablet f395b89623614d9990bc06b00ad24021 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:34:10 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34830
I20260927 02:34:10.464911 8179 tablet_service.cc:1514] Processing DeleteTablet for tablet f395b89623614d9990bc06b00ad24021 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:34:10 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41258
I20260927 02:34:10.465059 8437 tablet_replica.cc:330] T f395b89623614d9990bc06b00ad24021 P dfcc410419d7462193cfcc270fc06b54: stopping tablet replica
I20260927 02:34:10.465090 8436 tablet_replica.cc:330] T f395b89623614d9990bc06b00ad24021 P 619d0542fa744235954689b0f8badebb: stopping tablet replica
I20260927 02:34:10.465170 8437 raft_consensus.cc:2229] T f395b89623614d9990bc06b00ad24021 P dfcc410419d7462193cfcc270fc06b54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:34:10.465183 8436 raft_consensus.cc:2229] T f395b89623614d9990bc06b00ad24021 P 619d0542fa744235954689b0f8badebb [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:34:10.465282 8437 raft_consensus.cc:2258] T f395b89623614d9990bc06b00ad24021 P dfcc410419d7462193cfcc270fc06b54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:34:10.465332 8436 raft_consensus.cc:2258] T f395b89623614d9990bc06b00ad24021 P 619d0542fa744235954689b0f8badebb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:34:10.465487 8435 tablet_replica.cc:330] T f395b89623614d9990bc06b00ad24021 P cc8fdd96bba24e3682c3ba405fcc0a45: stopping tablet replica
I20260927 02:34:10.465590 8435 raft_consensus.cc:2229] T f395b89623614d9990bc06b00ad24021 P cc8fdd96bba24e3682c3ba405fcc0a45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:34:10.465705 8435 raft_consensus.cc:2258] T f395b89623614d9990bc06b00ad24021 P cc8fdd96bba24e3682c3ba405fcc0a45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:34:10.466033 7902 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35094:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 02:34:10.466131 8437 ts_tablet_manager.cc:1905] T f395b89623614d9990bc06b00ad24021 P dfcc410419d7462193cfcc270fc06b54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:34:10.466266 8436 ts_tablet_manager.cc:1905] T f395b89623614d9990bc06b00ad24021 P 619d0542fa744235954689b0f8badebb: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 02:34:10.466394 7902 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:34:10.467306 8435 ts_tablet_manager.cc:1905] T f395b89623614d9990bc06b00ad24021 P cc8fdd96bba24e3682c3ba405fcc0a45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:34:10.468384 8436 ts_tablet_manager.cc:1918] T f395b89623614d9990bc06b00ad24021 P 619d0542fa744235954689b0f8badebb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:34:10.468384 8437 ts_tablet_manager.cc:1918] T f395b89623614d9990bc06b00ad24021 P dfcc410419d7462193cfcc270fc06b54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:34:10.468453 8437 log.cc:1196] T f395b89623614d9990bc06b00ad24021 P dfcc410419d7462193cfcc270fc06b54: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-0/wal/wals/f395b89623614d9990bc06b00ad24021
I20260927 02:34:10.468464 8436 log.cc:1196] T f395b89623614d9990bc06b00ad24021 P 619d0542fa744235954689b0f8badebb: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-1/wal/wals/f395b89623614d9990bc06b00ad24021
I20260927 02:34:10.468684 8437 ts_tablet_manager.cc:1939] T f395b89623614d9990bc06b00ad24021 P dfcc410419d7462193cfcc270fc06b54: Deleting consensus metadata
I20260927 02:34:10.468694 8436 ts_tablet_manager.cc:1939] T f395b89623614d9990bc06b00ad24021 P 619d0542fa744235954689b0f8badebb: Deleting consensus metadata
I20260927 02:34:10.469286 7886 catalog_manager.cc:4841] TS dfcc410419d7462193cfcc270fc06b54 (127.0.98.1:35293): tablet f395b89623614d9990bc06b00ad24021 (table test-table [id=c6456644988240c1ac2a0955b643c30f]) successfully deleted
I20260927 02:34:10.469286 7889 catalog_manager.cc:4841] TS 619d0542fa744235954689b0f8badebb (127.0.98.2:43319): tablet f395b89623614d9990bc06b00ad24021 (table test-table [id=c6456644988240c1ac2a0955b643c30f]) successfully deleted
I20260927 02:34:10.469602 8435 ts_tablet_manager.cc:1918] T f395b89623614d9990bc06b00ad24021 P cc8fdd96bba24e3682c3ba405fcc0a45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:34:10.469674 8435 log.cc:1196] T f395b89623614d9990bc06b00ad24021 P cc8fdd96bba24e3682c3ba405fcc0a45: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-2/wal/wals/f395b89623614d9990bc06b00ad24021
I20260927 02:34:10.469904 8435 ts_tablet_manager.cc:1939] T f395b89623614d9990bc06b00ad24021 P cc8fdd96bba24e3682c3ba405fcc0a45: Deleting consensus metadata
I20260927 02:34:10.470567 7888 catalog_manager.cc:4841] TS cc8fdd96bba24e3682c3ba405fcc0a45 (127.0.98.3:45319): tablet f395b89623614d9990bc06b00ad24021 (table test-table [id=c6456644988240c1ac2a0955b643c30f]) successfully deleted
I20260927 02:34:10.470837 8179 tablet_service.cc:1467] Processing CreateTablet for tablet 9ec126f8ab2c40abb31fdc8203ec9256 (DEFAULT_TABLE table=test-table [id=60b7cc9a77c54aedacaec63f986b8fe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:34:10.470837 8319 tablet_service.cc:1467] Processing CreateTablet for tablet 9ec126f8ab2c40abb31fdc8203ec9256 (DEFAULT_TABLE table=test-table [id=60b7cc9a77c54aedacaec63f986b8fe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:34:10.470839 8039 tablet_service.cc:1467] Processing CreateTablet for tablet 9ec126f8ab2c40abb31fdc8203ec9256 (DEFAULT_TABLE table=test-table [id=60b7cc9a77c54aedacaec63f986b8fe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:34:10.471014 8179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ec126f8ab2c40abb31fdc8203ec9256. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:34:10.471043 8319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ec126f8ab2c40abb31fdc8203ec9256. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:34:10.471043 8039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ec126f8ab2c40abb31fdc8203ec9256. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:34:10.472532 8413 tablet_bootstrap.cc:492] T 9ec126f8ab2c40abb31fdc8203ec9256 P 619d0542fa744235954689b0f8badebb: Bootstrap starting.
I20260927 02:34:10.472972 8415 tablet_bootstrap.cc:492] T 9ec126f8ab2c40abb31fdc8203ec9256 P cc8fdd96bba24e3682c3ba405fcc0a45: Bootstrap starting.
I20260927 02:34:10.473019 8414 tablet_bootstrap.cc:492] T 9ec126f8ab2c40abb31fdc8203ec9256 P dfcc410419d7462193cfcc270fc06b54: Bootstrap starting.
I20260927 02:34:10.473312 8413 tablet_bootstrap.cc:654] T 9ec126f8ab2c40abb31fdc8203ec9256 P 619d0542fa744235954689b0f8badebb: Neither blocks nor log segments found. Creating new log.
I20260927 02:34:10.473729 8415 tablet_bootstrap.cc:654] T 9ec126f8ab2c40abb31fdc8203ec9256 P cc8fdd96bba24e3682c3ba405fcc0a45: Neither blocks nor log segments found. Creating new log.
I20260927 02:34:10.473737 8414 tablet_bootstrap.cc:654] T 9ec126f8ab2c40abb31fdc8203ec9256 P dfcc410419d7462193cfcc270fc06b54: Neither blocks nor log segments found. Creating new log.
I20260927 02:34:10.474331 8413 tablet_bootstrap.cc:492] T 9ec126f8ab2c40abb31fdc8203ec9256 P 619d0542fa744235954689b0f8badebb: No bootstrap required, opened a new log
I20260927 02:34:10.474414 8413 ts_tablet_manager.cc:1397] T 9ec126f8ab2c40abb31fdc8203ec9256 P 619d0542fa744235954689b0f8badebb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:34:10.474691 8415 tablet_bootstrap.cc:492] T 9ec126f8ab2c40abb31fdc8203ec9256 P cc8fdd96bba24e3682c3ba405fcc0a45: No bootstrap required, opened a new log
I20260927 02:34:10.474771 8415 ts_tablet_manager.cc:1397] T 9ec126f8ab2c40abb31fdc8203ec9256 P cc8fdd96bba24e3682c3ba405fcc0a45: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:34:10.474815 8413 raft_consensus.cc:348] T 9ec126f8ab2c40abb31fdc8203ec9256 P 619d0542fa744235954689b0f8badebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } } peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } }
I20260927 02:34:10.474922 8413 raft_consensus.cc:374] T 9ec126f8ab2c40abb31fdc8203ec9256 P 619d0542fa744235954689b0f8badebb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:34:10.474952 8413 raft_consensus.cc:729] T 9ec126f8ab2c40abb31fdc8203ec9256 P 619d0542fa744235954689b0f8badebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 619d0542fa744235954689b0f8badebb, State: Initialized, Role: FOLLOWER
I20260927 02:34:10.475065 8413 consensus_queue.cc:260] T 9ec126f8ab2c40abb31fdc8203ec9256 P 619d0542fa744235954689b0f8badebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } } peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } }
I20260927 02:34:10.475123 8415 raft_consensus.cc:348] T 9ec126f8ab2c40abb31fdc8203ec9256 P cc8fdd96bba24e3682c3ba405fcc0a45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } } peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } }
I20260927 02:34:10.475241 8415 raft_consensus.cc:374] T 9ec126f8ab2c40abb31fdc8203ec9256 P cc8fdd96bba24e3682c3ba405fcc0a45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:34:10.475278 8415 raft_consensus.cc:729] T 9ec126f8ab2c40abb31fdc8203ec9256 P cc8fdd96bba24e3682c3ba405fcc0a45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8fdd96bba24e3682c3ba405fcc0a45, State: Initialized, Role: FOLLOWER
I20260927 02:34:10.475322 8413 ts_tablet_manager.cc:1428] T 9ec126f8ab2c40abb31fdc8203ec9256 P 619d0542fa744235954689b0f8badebb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 02:34:10.475391 8415 consensus_queue.cc:260] T 9ec126f8ab2c40abb31fdc8203ec9256 P cc8fdd96bba24e3682c3ba405fcc0a45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } } peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } }
I20260927 02:34:10.475631 8415 ts_tablet_manager.cc:1428] T 9ec126f8ab2c40abb31fdc8203ec9256 P cc8fdd96bba24e3682c3ba405fcc0a45: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:34:10.476085 8414 tablet_bootstrap.cc:492] T 9ec126f8ab2c40abb31fdc8203ec9256 P dfcc410419d7462193cfcc270fc06b54: No bootstrap required, opened a new log
I20260927 02:34:10.476171 8414 ts_tablet_manager.cc:1397] T 9ec126f8ab2c40abb31fdc8203ec9256 P dfcc410419d7462193cfcc270fc06b54: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260927 02:34:10.476588 8414 raft_consensus.cc:348] T 9ec126f8ab2c40abb31fdc8203ec9256 P dfcc410419d7462193cfcc270fc06b54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } } peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } }
I20260927 02:34:10.476697 8414 raft_consensus.cc:374] T 9ec126f8ab2c40abb31fdc8203ec9256 P dfcc410419d7462193cfcc270fc06b54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:34:10.476729 8414 raft_consensus.cc:729] T 9ec126f8ab2c40abb31fdc8203ec9256 P dfcc410419d7462193cfcc270fc06b54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfcc410419d7462193cfcc270fc06b54, State: Initialized, Role: FOLLOWER
I20260927 02:34:10.476828 8414 consensus_queue.cc:260] T 9ec126f8ab2c40abb31fdc8203ec9256 P dfcc410419d7462193cfcc270fc06b54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } } peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } }
I20260927 02:34:10.477072 8414 ts_tablet_manager.cc:1428] T 9ec126f8ab2c40abb31fdc8203ec9256 P dfcc410419d7462193cfcc270fc06b54: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 02:34:10.732908 8469 raft_consensus.cc:482] T 9ec126f8ab2c40abb31fdc8203ec9256 P 619d0542fa744235954689b0f8badebb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:34:10.733072 8469 raft_consensus.cc:504] T 9ec126f8ab2c40abb31fdc8203ec9256 P 619d0542fa744235954689b0f8badebb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } } peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } }
I20260927 02:34:10.733505 8469 leader_election.cc:288] T 9ec126f8ab2c40abb31fdc8203ec9256 P 619d0542fa744235954689b0f8badebb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfcc410419d7462193cfcc270fc06b54 (127.0.98.1:35293), cc8fdd96bba24e3682c3ba405fcc0a45 (127.0.98.3:45319)
I20260927 02:34:10.733970 8339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ec126f8ab2c40abb31fdc8203ec9256" candidate_uuid: "619d0542fa744235954689b0f8badebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" is_pre_election: true
I20260927 02:34:10.733970 8058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ec126f8ab2c40abb31fdc8203ec9256" candidate_uuid: "619d0542fa744235954689b0f8badebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfcc410419d7462193cfcc270fc06b54" is_pre_election: true
I20260927 02:34:10.734131 8339 raft_consensus.cc:2454] T 9ec126f8ab2c40abb31fdc8203ec9256 P cc8fdd96bba24e3682c3ba405fcc0a45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 619d0542fa744235954689b0f8badebb in term 0.
I20260927 02:34:10.734146 8058 raft_consensus.cc:2454] T 9ec126f8ab2c40abb31fdc8203ec9256 P dfcc410419d7462193cfcc270fc06b54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 619d0542fa744235954689b0f8badebb in term 0.
I20260927 02:34:10.734393 8132 leader_election.cc:302] T 9ec126f8ab2c40abb31fdc8203ec9256 P 619d0542fa744235954689b0f8badebb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 619d0542fa744235954689b0f8badebb, cc8fdd96bba24e3682c3ba405fcc0a45; no voters:
I20260927 02:34:10.734568 8469 raft_consensus.cc:2781] T 9ec126f8ab2c40abb31fdc8203ec9256 P 619d0542fa744235954689b0f8badebb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:34:10.734628 8469 raft_consensus.cc:482] T 9ec126f8ab2c40abb31fdc8203ec9256 P 619d0542fa744235954689b0f8badebb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:34:10.734674 8469 raft_consensus.cc:3037] T 9ec126f8ab2c40abb31fdc8203ec9256 P 619d0542fa744235954689b0f8badebb [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:34:10.735591 8469 raft_consensus.cc:504] T 9ec126f8ab2c40abb31fdc8203ec9256 P 619d0542fa744235954689b0f8badebb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } } peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } }
I20260927 02:34:10.735873 8469 leader_election.cc:288] T 9ec126f8ab2c40abb31fdc8203ec9256 P 619d0542fa744235954689b0f8badebb [CANDIDATE]: Term 1 election: Requested vote from peers dfcc410419d7462193cfcc270fc06b54 (127.0.98.1:35293), cc8fdd96bba24e3682c3ba405fcc0a45 (127.0.98.3:45319)
I20260927 02:34:10.736202 8058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ec126f8ab2c40abb31fdc8203ec9256" candidate_uuid: "619d0542fa744235954689b0f8badebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfcc410419d7462193cfcc270fc06b54"
I20260927 02:34:10.736222 8339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ec126f8ab2c40abb31fdc8203ec9256" candidate_uuid: "619d0542fa744235954689b0f8badebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45"
I20260927 02:34:10.736330 8058 raft_consensus.cc:3037] T 9ec126f8ab2c40abb31fdc8203ec9256 P dfcc410419d7462193cfcc270fc06b54 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:34:10.736330 8339 raft_consensus.cc:3037] T 9ec126f8ab2c40abb31fdc8203ec9256 P cc8fdd96bba24e3682c3ba405fcc0a45 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:34:10.737416 8058 raft_consensus.cc:2454] T 9ec126f8ab2c40abb31fdc8203ec9256 P dfcc410419d7462193cfcc270fc06b54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 619d0542fa744235954689b0f8badebb in term 1.
I20260927 02:34:10.737419 8339 raft_consensus.cc:2454] T 9ec126f8ab2c40abb31fdc8203ec9256 P cc8fdd96bba24e3682c3ba405fcc0a45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 619d0542fa744235954689b0f8badebb in term 1.
I20260927 02:34:10.737684 8132 leader_election.cc:302] T 9ec126f8ab2c40abb31fdc8203ec9256 P 619d0542fa744235954689b0f8badebb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 619d0542fa744235954689b0f8badebb, cc8fdd96bba24e3682c3ba405fcc0a45, dfcc410419d7462193cfcc270fc06b54; no voters:
I20260927 02:34:10.737866 8469 raft_consensus.cc:2781] T 9ec126f8ab2c40abb31fdc8203ec9256 P 619d0542fa744235954689b0f8badebb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:34:10.737948 8469 raft_consensus.cc:686] T 9ec126f8ab2c40abb31fdc8203ec9256 P 619d0542fa744235954689b0f8badebb [term 1 LEADER]: Becoming Leader. State: Replica: 619d0542fa744235954689b0f8badebb, State: Running, Role: LEADER
I20260927 02:34:10.738094 8469 consensus_queue.cc:237] T 9ec126f8ab2c40abb31fdc8203ec9256 P 619d0542fa744235954689b0f8badebb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } } peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } }
I20260927 02:34:10.739526 7899 catalog_manager.cc:5495] T 9ec126f8ab2c40abb31fdc8203ec9256 P 619d0542fa744235954689b0f8badebb reported cstate change: term changed from 0 to 1, leader changed from <none> to 619d0542fa744235954689b0f8badebb (127.0.98.2). New cstate: current_term: 1 leader_uuid: "619d0542fa744235954689b0f8badebb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "619d0542fa744235954689b0f8badebb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 } health_report { overall_health: UNKNOWN } } }
I20260927 02:34:10.778671 8339 raft_consensus.cc:1261] T 9ec126f8ab2c40abb31fdc8203ec9256 P cc8fdd96bba24e3682c3ba405fcc0a45 [term 1 FOLLOWER]: Refusing update from remote peer 619d0542fa744235954689b0f8badebb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:34:10.778725 8058 raft_consensus.cc:1261] T 9ec126f8ab2c40abb31fdc8203ec9256 P dfcc410419d7462193cfcc270fc06b54 [term 1 FOLLOWER]: Refusing update from remote peer 619d0542fa744235954689b0f8badebb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:34:10.779075 8423 consensus_queue.cc:1035] T 9ec126f8ab2c40abb31fdc8203ec9256 P 619d0542fa744235954689b0f8badebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfcc410419d7462193cfcc270fc06b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:34:10.779222 8469 consensus_queue.cc:1035] T 9ec126f8ab2c40abb31fdc8203ec9256 P 619d0542fa744235954689b0f8badebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8fdd96bba24e3682c3ba405fcc0a45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:34:10.807410 7902 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:34:10.807550 7902 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:34:10.809312 7902 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e302084e70d14287803dacb09876350e: Sending DeleteTablet for 3 replicas of tablet 9ec126f8ab2c40abb31fdc8203ec9256
I20260927 02:34:10.809888 8179 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ec126f8ab2c40abb31fdc8203ec9256 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:34:10 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41258
I20260927 02:34:10.809888 8319 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ec126f8ab2c40abb31fdc8203ec9256 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:34:10 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34830
I20260927 02:34:10.810050 8435 tablet_replica.cc:330] T 9ec126f8ab2c40abb31fdc8203ec9256 P cc8fdd96bba24e3682c3ba405fcc0a45: stopping tablet replica
I20260927 02:34:10.810073 8436 tablet_replica.cc:330] T 9ec126f8ab2c40abb31fdc8203ec9256 P 619d0542fa744235954689b0f8badebb: stopping tablet replica
I20260927 02:34:10.810048 8039 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ec126f8ab2c40abb31fdc8203ec9256 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:34:10 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60972
I20260927 02:34:10.810155 8436 raft_consensus.cc:2229] T 9ec126f8ab2c40abb31fdc8203ec9256 P 619d0542fa744235954689b0f8badebb [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:34:10.810151 8435 raft_consensus.cc:2229] T 9ec126f8ab2c40abb31fdc8203ec9256 P cc8fdd96bba24e3682c3ba405fcc0a45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:34:10.810227 8437 tablet_replica.cc:330] T 9ec126f8ab2c40abb31fdc8203ec9256 P dfcc410419d7462193cfcc270fc06b54: stopping tablet replica
I20260927 02:34:10.810248 8435 raft_consensus.cc:2258] T 9ec126f8ab2c40abb31fdc8203ec9256 P cc8fdd96bba24e3682c3ba405fcc0a45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:34:10.810317 8436 raft_consensus.cc:2258] T 9ec126f8ab2c40abb31fdc8203ec9256 P 619d0542fa744235954689b0f8badebb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:34:10.810333 8437 raft_consensus.cc:2229] T 9ec126f8ab2c40abb31fdc8203ec9256 P dfcc410419d7462193cfcc270fc06b54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:34:10.810428 8437 raft_consensus.cc:2258] T 9ec126f8ab2c40abb31fdc8203ec9256 P dfcc410419d7462193cfcc270fc06b54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:34:10.810604 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 7974
I20260927 02:34:10.810953 8436 ts_tablet_manager.cc:1905] T 9ec126f8ab2c40abb31fdc8203ec9256 P 619d0542fa744235954689b0f8badebb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:34:10.811589 8435 ts_tablet_manager.cc:1905] T 9ec126f8ab2c40abb31fdc8203ec9256 P cc8fdd96bba24e3682c3ba405fcc0a45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:34:10.812867 8436 ts_tablet_manager.cc:1918] T 9ec126f8ab2c40abb31fdc8203ec9256 P 619d0542fa744235954689b0f8badebb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:34:10.813944 8435 ts_tablet_manager.cc:1918] T 9ec126f8ab2c40abb31fdc8203ec9256 P cc8fdd96bba24e3682c3ba405fcc0a45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:34:10.814006 8435 log.cc:1196] T 9ec126f8ab2c40abb31fdc8203ec9256 P cc8fdd96bba24e3682c3ba405fcc0a45: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-2/wal/wals/9ec126f8ab2c40abb31fdc8203ec9256
I20260927 02:34:10.814211 8435 ts_tablet_manager.cc:1939] T 9ec126f8ab2c40abb31fdc8203ec9256 P cc8fdd96bba24e3682c3ba405fcc0a45: Deleting consensus metadata
W20260927 02:34:10.817294 7886 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35293 (error 108)
I20260927 02:34:10.817430 7888 catalog_manager.cc:4841] TS cc8fdd96bba24e3682c3ba405fcc0a45 (127.0.98.3:45319): tablet 9ec126f8ab2c40abb31fdc8203ec9256 (table test-table [id=60b7cc9a77c54aedacaec63f986b8fe9]) successfully deleted
I20260927 02:34:10.817458 8436 log.cc:1196] T 9ec126f8ab2c40abb31fdc8203ec9256 P 619d0542fa744235954689b0f8badebb: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476410933508-392-0/minicluster-data/ts-1/wal/wals/9ec126f8ab2c40abb31fdc8203ec9256
I20260927 02:34:10.817734 8436 ts_tablet_manager.cc:1939] T 9ec126f8ab2c40abb31fdc8203ec9256 P 619d0542fa744235954689b0f8badebb: Deleting consensus metadata
I20260927 02:34:10.818066 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 8114
I20260927 02:34:10.819051 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:10.818328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35293 (local address 127.0.0.1:37806)
0927 02:34:10.818892 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:10.818959 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":47,"threads_started":1}
W20260927 02:34:10.819177 7886 catalog_manager.cc:4584] TS dfcc410419d7462193cfcc270fc06b54 (127.0.98.1:35293): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9ec126f8ab2c40abb31fdc8203ec9256: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35293: connect: Connection refused (error 111)
I20260927 02:34:10.824332 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 8254
I20260927 02:34:10.825280 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:10.824480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43319 (local address 127.0.0.1:50378)
0927 02:34:10.825162 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:10.825210 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:43319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":612}
I20260927 02:34:10.829500 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 7870
2026-09-27T02:34:10Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3208 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476410933508-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:34:10 dist-test-slave-grlk krb5kdc[8504](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:34:10 dist-test-slave-grlk krb5kdc[8504](info): set up 2 sockets
Sep 27 02:34:10 dist-test-slave-grlk krb5kdc[8504](info): commencing operation
krb5kdc: starting...
W20260927 02:34:11.773236 392 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476410933508-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476410933508-392-0/krb5kdc/test-user.keytab.
Sep 27 02:34:11 dist-test-slave-grlk krb5kdc[8504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:34:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:34:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:34:11.883555 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35673
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:34:11.987051 8517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:34:11.987293 8517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:34:11.987332 8517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:34:11.991245 8517 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:34:11.991302 8517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:34:11.991318 8517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:34:11.991333 8517 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:34:11.991346 8517 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:34:11.996490 8517 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:34:11.997560 8517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:34:11.998564 8517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:34:12.006501 8522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:12.006537 8525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:12.006637 8517 server_base.cc:1034] running on GCE node
W20260927 02:34:12.006590 8523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:12.007259 8517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:34:12.008327 8517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:34:12.009478 8517 hybrid_clock.cc:648] HybridClock initialized: now 1790476452009462 us; error 41 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T02:34:12Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1192 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:34:12 dist-test-slave-grlk krb5kdc[8530](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:34:12 dist-test-slave-grlk krb5kdc[8530](info): set up 2 sockets
Sep 27 02:34:12 dist-test-slave-grlk krb5kdc[8530](info): commencing operation
krb5kdc: starting...
W20260927 02:34:12.965422 392 mini_kdc.cc:121] Time spent starting KDC: real 0.922s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/test-user.keytab.
Sep 27 02:34:13 dist-test-slave-grlk krb5kdc[8530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:34:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:34:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:34:13.081686 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37517
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37517
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:34:13.190192 8543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:34:13.190461 8543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:34:13.190502 8543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:34:13.194501 8543 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:34:13.194559 8543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:34:13.194578 8543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:34:13.194593 8543 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:34:13.194607 8543 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:34:13.199836 8543 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:34:13.200830 8543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:34:13.201799 8543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:34:13.209728 8549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:13.209749 8551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:13.209761 8548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:13.209856 8543 server_base.cc:1034] running on GCE node
I20260927 02:34:13.210510 8543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:34:13.211535 8543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:34:13.212682 8543 hybrid_clock.cc:648] HybridClock initialized: now 1790476453212659 us; error 33 us; skew 500 ppm
Sep 27 02:34:13 dist-test-slave-grlk krb5kdc[8530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:34:13.215164 8543 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:34:13.215981 8543 webserver.cc:460] Webserver started at http://127.0.98.62:45699/ using document root <none> and password file <none>
I20260927 02:34:13.216490 8543 fs_manager.cc:359] Metadata directory not provided
I20260927 02:34:13.216548 8543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:34:13.216722 8543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:34:13.218626 8543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "5f77be4f93cf401e8bafd3c2af25ab5f"
format_stamp: "Formatted at 2026-09-27 02:34:13 on dist-test-slave-grlk"
server_key: "89d7cb041e8531be4e521044e911dd0a"
server_key_iv: "7f8c78ab9335401bf4741429223b96ab"
server_key_version: "encryptionkey@0"
I20260927 02:34:13.219113 8543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "5f77be4f93cf401e8bafd3c2af25ab5f"
format_stamp: "Formatted at 2026-09-27 02:34:13 on dist-test-slave-grlk"
server_key: "89d7cb041e8531be4e521044e911dd0a"
server_key_iv: "7f8c78ab9335401bf4741429223b96ab"
server_key_version: "encryptionkey@0"
I20260927 02:34:13.222493 8543 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:34:13.224635 8558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:13.225782 8543 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:34:13.225894 8543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "5f77be4f93cf401e8bafd3c2af25ab5f"
format_stamp: "Formatted at 2026-09-27 02:34:13 on dist-test-slave-grlk"
server_key: "89d7cb041e8531be4e521044e911dd0a"
server_key_iv: "7f8c78ab9335401bf4741429223b96ab"
server_key_version: "encryptionkey@0"
I20260927 02:34:13.225982 8543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:34:13.243077 8543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:34:13.245726 8543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:34:13.245894 8543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:34:13.252194 8543 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37517
I20260927 02:34:13.252213 8610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37517 every 8 connection(s)
I20260927 02:34:13.253114 8543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:34:13.255661 8611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:34:13.257094 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 8543
I20260927 02:34:13.257206 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:34:13.257458 392 external_mini_cluster.cc:1428] Setting key a3fde12e34af1b9464783a6ec33bf720
I20260927 02:34:13.261698 8611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f77be4f93cf401e8bafd3c2af25ab5f: Bootstrap starting.
I20260927 02:34:13.264038 8611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f77be4f93cf401e8bafd3c2af25ab5f: Neither blocks nor log segments found. Creating new log.
I20260927 02:34:13.264854 8611 log.cc:824] T 00000000000000000000000000000000 P 5f77be4f93cf401e8bafd3c2af25ab5f: Log is configured to *not* fsync() on all Append() calls
Sep 27 02:34:13 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:34:13.266788 8611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f77be4f93cf401e8bafd3c2af25ab5f: No bootstrap required, opened a new log
I20260927 02:34:13.271047 8611 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f77be4f93cf401e8bafd3c2af25ab5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f77be4f93cf401e8bafd3c2af25ab5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37517 } }
I20260927 02:34:13.271301 8611 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f77be4f93cf401e8bafd3c2af25ab5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:34:13.271353 8611 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f77be4f93cf401e8bafd3c2af25ab5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f77be4f93cf401e8bafd3c2af25ab5f, State: Initialized, Role: FOLLOWER
I20260927 02:34:13.271732 8614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:13.258725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49198 (local address 127.0.98.62:37517)
0927 02:34:13.259108 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:13.259135 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:13.259176 (+ 41us) server_negotiation.cc:407] Connection header received
0927 02:34:13.259934 (+ 758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:13.259956 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:13.260356 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:13.260696 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:13.261335 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:13.262707 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:13.263524 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:13.264168 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:13.264321 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:13.266839 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:13.266845 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:13.266860 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:13.266888 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:13.268655 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:13.269174 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:13.269181 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:13.269186 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:13.269321 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:13.269620 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:13.269623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:13.269624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:13.270135 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:13.270327 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:13.270518 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 02:34:13.270805 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":100,"threads_started":1}
I20260927 02:34:13.271809 8611 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f77be4f93cf401e8bafd3c2af25ab5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f77be4f93cf401e8bafd3c2af25ab5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37517 } }
I20260927 02:34:13.271960 8611 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f77be4f93cf401e8bafd3c2af25ab5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:34:13.272023 8611 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f77be4f93cf401e8bafd3c2af25ab5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:34:13.272110 8611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f77be4f93cf401e8bafd3c2af25ab5f [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:34:13.273166 8611 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f77be4f93cf401e8bafd3c2af25ab5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f77be4f93cf401e8bafd3c2af25ab5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37517 } }
I20260927 02:34:13.273476 8611 leader_election.cc:302] T 00000000000000000000000000000000 P 5f77be4f93cf401e8bafd3c2af25ab5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f77be4f93cf401e8bafd3c2af25ab5f; no voters:
I20260927 02:34:13.273790 8611 leader_election.cc:288] T 00000000000000000000000000000000 P 5f77be4f93cf401e8bafd3c2af25ab5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:34:13.273871 8616 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f77be4f93cf401e8bafd3c2af25ab5f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:34:13.274101 8616 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f77be4f93cf401e8bafd3c2af25ab5f [term 1 LEADER]: Becoming Leader. State: Replica: 5f77be4f93cf401e8bafd3c2af25ab5f, State: Running, Role: LEADER
I20260927 02:34:13.274385 8616 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f77be4f93cf401e8bafd3c2af25ab5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f77be4f93cf401e8bafd3c2af25ab5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37517 } }
I20260927 02:34:13.275049 8611 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f77be4f93cf401e8bafd3c2af25ab5f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:34:13.275923 8618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f77be4f93cf401e8bafd3c2af25ab5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f77be4f93cf401e8bafd3c2af25ab5f. Latest consensus state: current_term: 1 leader_uuid: "5f77be4f93cf401e8bafd3c2af25ab5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f77be4f93cf401e8bafd3c2af25ab5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37517 } } }
I20260927 02:34:13.276028 8618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f77be4f93cf401e8bafd3c2af25ab5f [sys.catalog]: This master's current role is: LEADER
I20260927 02:34:13.276345 8617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f77be4f93cf401e8bafd3c2af25ab5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f77be4f93cf401e8bafd3c2af25ab5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f77be4f93cf401e8bafd3c2af25ab5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37517 } } }
I20260927 02:34:13.276423 8625 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:34:13.276489 8617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f77be4f93cf401e8bafd3c2af25ab5f [sys.catalog]: This master's current role is: LEADER
I20260927 02:34:13.279078 8625 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:34:13.283829 8625 catalog_manager.cc:1347] Generated new cluster ID: 617f15c67d844b3487e87a10737fb589
I20260927 02:34:13.283878 8625 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:34:13.289225 8625 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:34:13.290036 8625 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:34:13.295863 8625 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f77be4f93cf401e8bafd3c2af25ab5f: Generated new TSK 0
I20260927 02:34:13.296422 8625 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:34:13.328531 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37517
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:34:13.432514 8637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:34:13.432745 8637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:34:13.432785 8637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:34:13.436735 8637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:34:13.436787 8637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:34:13.436841 8637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:34:13.441744 8637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:34:13.442615 8637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:34:13.443529 8637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:34:13.452086 8642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:13.452086 8643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:13.452253 8637 server_base.cc:1034] running on GCE node
W20260927 02:34:13.452080 8645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:13.452615 8637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:34:13.453207 8637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:34:13.454371 8637 hybrid_clock.cc:648] HybridClock initialized: now 1790476453454345 us; error 35 us; skew 500 ppm
Sep 27 02:34:13 dist-test-slave-grlk krb5kdc[8530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:34:13.457425 8637 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:34:13.458206 8637 webserver.cc:460] Webserver started at http://127.0.98.1:44349/ using document root <none> and password file <none>
I20260927 02:34:13.458691 8637 fs_manager.cc:359] Metadata directory not provided
I20260927 02:34:13.458747 8637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:34:13.458932 8637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:34:13.460800 8637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-0/data/instance:
uuid: "1f4a36d8def143e2aeb0d9ae9a1815ea"
format_stamp: "Formatted at 2026-09-27 02:34:13 on dist-test-slave-grlk"
server_key: "05aa757df543dad642d7117101dec484"
server_key_iv: "731e21ec09f1679ca041f6619c4fcfd0"
server_key_version: "encryptionkey@0"
I20260927 02:34:13.461337 8637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1f4a36d8def143e2aeb0d9ae9a1815ea"
format_stamp: "Formatted at 2026-09-27 02:34:13 on dist-test-slave-grlk"
server_key: "05aa757df543dad642d7117101dec484"
server_key_iv: "731e21ec09f1679ca041f6619c4fcfd0"
server_key_version: "encryptionkey@0"
I20260927 02:34:13.464577 8637 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:34:13.466717 8652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:13.467839 8637 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:34:13.467935 8637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-0/wal
uuid: "1f4a36d8def143e2aeb0d9ae9a1815ea"
format_stamp: "Formatted at 2026-09-27 02:34:13 on dist-test-slave-grlk"
server_key: "05aa757df543dad642d7117101dec484"
server_key_iv: "731e21ec09f1679ca041f6619c4fcfd0"
server_key_version: "encryptionkey@0"
I20260927 02:34:13.468024 8637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:34:13.479738 8637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:34:13.482511 8637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:34:13.482666 8637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:34:13.483166 8637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:34:13.483930 8637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:34:13.483973 8637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:13.484021 8637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:34:13.484050 8637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:13.492082 8637 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45799
I20260927 02:34:13.492113 8765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45799 every 8 connection(s)
I20260927 02:34:13.492901 8637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:34:13.493973 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 8637
I20260927 02:34:13.494078 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:34:13.494375 392 external_mini_cluster.cc:1428] Setting key 2f805f57df69f0fc68fd3b5b2bf4eeae
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 27 02:34:13 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:34:13.508780 8614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:13.494673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47187 (local address 127.0.98.62:37517)
0927 02:34:13.494828 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:13.494832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:13.495654 (+ 822us) server_negotiation.cc:407] Connection header received
0927 02:34:13.497696 (+ 2042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:13.497699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:13.497759 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:13.497833 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:13.498979 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:13.499499 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:13.500608 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:13.500847 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:13.500953 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:13.505509 (+ 4556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:13.505512 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:13.505515 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:13.505540 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:13.506827 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:13.507313 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:13.507317 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:13.507318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:13.507405 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:13.507687 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:13.507690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:13.507692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:13.507908 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:13.508012 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:13.508471 (+ 459us) server_negotiation.cc:299] Negotiation successful
0927 02:34:13.508604 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 02:34:13.509629 8768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:13.495037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.1:47187)
0927 02:34:13.495509 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:13.495564 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:13.496851 (+ 1287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:13.498023 (+ 1172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:13.498030 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:13.498376 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:13.498832 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:13.498845 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:13.499618 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:13.499622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:13.500484 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:13.500493 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:13.501007 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:13.501011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:13.501171 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:13.501772 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:13.501788 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:13.505346 (+ 3558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:34:13.506928 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:13.506933 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:13.506945 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:13.507195 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:13.507488 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:13.507490 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:13.507491 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:13.507608 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:13.508005 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:34:13.508009 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:34:13.508246 (+ 237us) client_negotiation.cc:769] Sending connection context
0927 02:34:13.508395 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 02:34:13.508642 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":120,"threads_started":1}
I20260927 02:34:13.510859 8766 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37517
I20260927 02:34:13.511072 8766 heartbeater.cc:464] Registering TS with master...
I20260927 02:34:13.511533 8766 heartbeater.cc:511] Master 127.0.98.62:37517 requested a full tablet report, sending...
I20260927 02:34:13.512957 8575 ts_manager.cc:194] Registered new tserver with Master: 1f4a36d8def143e2aeb0d9ae9a1815ea (127.0.98.1:45799)
I20260927 02:34:13.514061 8575 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47187
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:34:13.519465 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37517
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:34:13.624186 8771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:34:13.624455 8771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:34:13.624516 8771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:34:13.628515 8771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:34:13.628574 8771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:34:13.628654 8771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:34:13.633876 8771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:34:13.634868 8771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:34:13.635854 8771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:34:13.644959 8779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:13.644990 8777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:13.644979 8776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:13.645231 8771 server_base.cc:1034] running on GCE node
I20260927 02:34:13.645685 8771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:34:13.646281 8771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:34:13.647461 8771 hybrid_clock.cc:648] HybridClock initialized: now 1790476453647441 us; error 29 us; skew 500 ppm
Sep 27 02:34:13 dist-test-slave-grlk krb5kdc[8530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:34:13.650030 8771 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:34:13.650837 8771 webserver.cc:460] Webserver started at http://127.0.98.2:35037/ using document root <none> and password file <none>
I20260927 02:34:13.651347 8771 fs_manager.cc:359] Metadata directory not provided
I20260927 02:34:13.651407 8771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:34:13.651584 8771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:34:13.653538 8771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-1/data/instance:
uuid: "fc9e2a8a98fc4690a0de61167a3269d8"
format_stamp: "Formatted at 2026-09-27 02:34:13 on dist-test-slave-grlk"
server_key: "e3179fb6b8233898541c2cd4f6156fb7"
server_key_iv: "7ab194b862ea2e3cd2a4728ae055c51e"
server_key_version: "encryptionkey@0"
I20260927 02:34:13.654066 8771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fc9e2a8a98fc4690a0de61167a3269d8"
format_stamp: "Formatted at 2026-09-27 02:34:13 on dist-test-slave-grlk"
server_key: "e3179fb6b8233898541c2cd4f6156fb7"
server_key_iv: "7ab194b862ea2e3cd2a4728ae055c51e"
server_key_version: "encryptionkey@0"
I20260927 02:34:13.657539 8771 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 02:34:13.659916 8786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:13.661158 8771 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:34:13.661260 8771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-1/wal
uuid: "fc9e2a8a98fc4690a0de61167a3269d8"
format_stamp: "Formatted at 2026-09-27 02:34:13 on dist-test-slave-grlk"
server_key: "e3179fb6b8233898541c2cd4f6156fb7"
server_key_iv: "7ab194b862ea2e3cd2a4728ae055c51e"
server_key_version: "encryptionkey@0"
I20260927 02:34:13.661353 8771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:34:13.687709 8771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:34:13.690773 8771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:34:13.690948 8771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:34:13.691504 8771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:34:13.692279 8771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:34:13.692324 8771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:13.692375 8771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:34:13.692474 8771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:13.700555 8771 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37045
I20260927 02:34:13.700579 8899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37045 every 8 connection(s)
I20260927 02:34:13.701452 8771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:34:13.705325 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 8771
I20260927 02:34:13.705461 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:34:13.705731 392 external_mini_cluster.cc:1428] Setting key c93db59c920912b27e3606fedc3f459d
Sep 27 02:34:13 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:34:13.715204 8614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:13.703137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52253 (local address 127.0.98.62:37517)
0927 02:34:13.703275 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:13.703278 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:13.704009 (+ 731us) server_negotiation.cc:407] Connection header received
0927 02:34:13.705425 (+ 1416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:13.705429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:13.705485 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:13.705567 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:13.706818 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:13.707403 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:13.708895 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:13.709253 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:13.709356 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:13.711774 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:13.711778 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:13.711781 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:13.711809 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:13.713360 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:13.713826 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:13.713830 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:13.713832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:13.713916 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:13.714205 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:13.714208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:13.714210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:13.714422 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:13.714521 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:13.714971 (+ 450us) server_negotiation.cc:299] Negotiation successful
0927 02:34:13.715086 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 02:34:13.716013 8902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:13.703465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.2:52253)
0927 02:34:13.703869 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:13.703909 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:13.705224 (+ 1315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:13.705753 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:13.705760 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:13.706123 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:13.706672 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:13.706691 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:13.707534 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:13.707537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:13.708786 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:13.708795 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:13.709382 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:13.709384 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:13.709518 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:13.710076 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:13.710100 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:13.711606 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:34:13.713463 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:13.713467 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:13.713478 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:13.713719 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:13.713999 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:13.714001 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:13.714002 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:13.714118 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:13.714507 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:34:13.714512 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:34:13.714740 (+ 228us) client_negotiation.cc:769] Sending connection context
0927 02:34:13.714908 (+ 168us) client_negotiation.cc:240] Negotiation successful
0927 02:34:13.715159 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":104,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 02:34:13.717260 8900 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37517
I20260927 02:34:13.717528 8900 heartbeater.cc:464] Registering TS with master...
I20260927 02:34:13.718024 8900 heartbeater.cc:511] Master 127.0.98.62:37517 requested a full tablet report, sending...
I20260927 02:34:13.719074 8575 ts_manager.cc:194] Registered new tserver with Master: fc9e2a8a98fc4690a0de61167a3269d8 (127.0.98.2:37045)
I20260927 02:34:13.719623 8575 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52253
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:34:13.731405 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37517
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:34:13.836227 8905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:34:13.836475 8905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:34:13.836530 8905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:34:13.840626 8905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:34:13.840687 8905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:34:13.840765 8905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:34:13.846073 8905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:34:13.847014 8905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:34:13.847946 8905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:34:13.856796 8913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:13.856822 8910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:13.856995 8905 server_base.cc:1034] running on GCE node
W20260927 02:34:13.856808 8911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:13.857564 8905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:34:13.858209 8905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:34:13.859370 8905 hybrid_clock.cc:648] HybridClock initialized: now 1790476453859354 us; error 25 us; skew 500 ppm
Sep 27 02:34:13 dist-test-slave-grlk krb5kdc[8530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:34:13.862017 8905 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:34:13.862788 8905 webserver.cc:460] Webserver started at http://127.0.98.3:37149/ using document root <none> and password file <none>
I20260927 02:34:13.863278 8905 fs_manager.cc:359] Metadata directory not provided
I20260927 02:34:13.863337 8905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:34:13.863497 8905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:34:13.865504 8905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-2/data/instance:
uuid: "32354cfc52be47acb3d5408c78d79455"
format_stamp: "Formatted at 2026-09-27 02:34:13 on dist-test-slave-grlk"
server_key: "34a9aff041cc5146647359f927c44f3f"
server_key_iv: "ed7bdf7b1a1a4c187ae269442b128f9d"
server_key_version: "encryptionkey@0"
I20260927 02:34:13.866078 8905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-2/wal/instance:
uuid: "32354cfc52be47acb3d5408c78d79455"
format_stamp: "Formatted at 2026-09-27 02:34:13 on dist-test-slave-grlk"
server_key: "34a9aff041cc5146647359f927c44f3f"
server_key_iv: "ed7bdf7b1a1a4c187ae269442b128f9d"
server_key_version: "encryptionkey@0"
I20260927 02:34:13.869711 8905 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:34:13.872212 8920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:13.873660 8905 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:34:13.873776 8905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-2/wal
uuid: "32354cfc52be47acb3d5408c78d79455"
format_stamp: "Formatted at 2026-09-27 02:34:13 on dist-test-slave-grlk"
server_key: "34a9aff041cc5146647359f927c44f3f"
server_key_iv: "ed7bdf7b1a1a4c187ae269442b128f9d"
server_key_version: "encryptionkey@0"
I20260927 02:34:13.873893 8905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:34:13.885082 8905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:34:13.887624 8905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:34:13.887779 8905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:34:13.888298 8905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:34:13.889056 8905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:34:13.889102 8905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:13.889143 8905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:34:13.889171 8905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:13.896845 8905 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43233
I20260927 02:34:13.896874 9033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43233 every 8 connection(s)
I20260927 02:34:13.897656 8905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 02:34:13 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:34:13.906664 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 8905
I20260927 02:34:13.906775 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:34:13.907001 392 external_mini_cluster.cc:1428] Setting key 1e8385da6be67b6c4e5973d30dee6515
I20260927 02:34:13.910099 8614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:13.899282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33155 (local address 127.0.98.62:37517)
0927 02:34:13.899409 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:13.899413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:13.900240 (+ 827us) server_negotiation.cc:407] Connection header received
0927 02:34:13.901635 (+ 1395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:13.901638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:13.901682 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:13.901754 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:13.902853 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:13.903362 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:13.904465 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:13.904671 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:13.904738 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:13.907004 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:13.907007 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:13.907009 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:13.907031 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:13.908465 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:13.908959 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:13.908963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:13.908965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:13.909046 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:13.909305 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:13.909307 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:13.909308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:13.909453 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:13.909518 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:13.909905 (+ 387us) server_negotiation.cc:299] Negotiation successful
0927 02:34:13.909986 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 02:34:13.910797 9036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:13.899615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.3:33155)
0927 02:34:13.900084 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:13.900125 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:13.901427 (+ 1302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:13.901905 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:13.901911 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:13.902252 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:13.902729 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:13.902743 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:13.903490 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:13.903493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:13.904369 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:13.904378 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:13.904774 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:13.904777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:13.904905 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:13.905481 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:13.905495 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:13.906855 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:34:13.908561 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:13.908565 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:13.908575 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:13.908872 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:13.909118 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:13.909121 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:13.909122 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:13.909246 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:13.909519 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:34:13.909523 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:34:13.909733 (+ 210us) client_negotiation.cc:769] Sending connection context
0927 02:34:13.909871 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 02:34:13.910092 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":106,"threads_started":1}
I20260927 02:34:13.911758 9034 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37517
I20260927 02:34:13.911957 9034 heartbeater.cc:464] Registering TS with master...
I20260927 02:34:13.912367 9034 heartbeater.cc:511] Master 127.0.98.62:37517 requested a full tablet report, sending...
I20260927 02:34:13.913264 8575 ts_manager.cc:194] Registered new tserver with Master: 32354cfc52be47acb3d5408c78d79455 (127.0.98.3:43233)
I20260927 02:34:13.913786 8575 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33155
I20260927 02:34:13.920286 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 02:34:13.987321 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 8637
I20260927 02:34:13.992192 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 8771
I20260927 02:34:13.996584 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 8905
I20260927 02:34:14.000794 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 8543
I20260927 02:34:14.005856 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37517
--webserver_interface=127.0.98.62
--webserver_port=45699
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37517
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:34:14.112004 9043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:34:14.112265 9043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:34:14.112321 9043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:34:14.116338 9043 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:34:14.116406 9043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:34:14.116433 9043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:34:14.116454 9043 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:34:14.116474 9043 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:34:14.121781 9043 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=45699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:34:14.122828 9043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:34:14.123818 9043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:34:14.132131 9051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:14.132210 9049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:14.132304 9043 server_base.cc:1034] running on GCE node
W20260927 02:34:14.132232 9048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:14.132810 9043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:34:14.133858 9043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:34:14.135017 9043 hybrid_clock.cc:648] HybridClock initialized: now 1790476454134987 us; error 34 us; skew 500 ppm
Sep 27 02:34:14 dist-test-slave-grlk krb5kdc[8530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:34:14.137553 9043 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:34:14.138388 9043 webserver.cc:460] Webserver started at http://127.0.98.62:45699/ using document root <none> and password file <none>
I20260927 02:34:14.138892 9043 fs_manager.cc:359] Metadata directory not provided
I20260927 02:34:14.138962 9043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:34:14.142575 9043 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:34:14.144513 9058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:14.145674 9043 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:34:14.145771 9043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "5f77be4f93cf401e8bafd3c2af25ab5f"
format_stamp: "Formatted at 2026-09-27 02:34:13 on dist-test-slave-grlk"
server_key: "89d7cb041e8531be4e521044e911dd0a"
server_key_iv: "7f8c78ab9335401bf4741429223b96ab"
server_key_version: "encryptionkey@0"
I20260927 02:34:14.146281 9043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:34:14.203557 9043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:34:14.206746 9043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:34:14.206928 9043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:34:14.213985 9043 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37517
I20260927 02:34:14.214004 9110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37517 every 8 connection(s)
I20260927 02:34:14.215014 9043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:34:14.219012 9111 sys_catalog.cc:263] Verifying existing consensus state
I20260927 02:34:14.220556 9111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f77be4f93cf401e8bafd3c2af25ab5f: Bootstrap starting.
I20260927 02:34:14.221745 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 9043
I20260927 02:34:14.222172 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45799
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37517
--webserver_port=44349
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:34:14.236102 9111 log.cc:824] T 00000000000000000000000000000000 P 5f77be4f93cf401e8bafd3c2af25ab5f: Log is configured to *not* fsync() on all Append() calls
I20260927 02:34:14.242393 9111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f77be4f93cf401e8bafd3c2af25ab5f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:34:14.242717 9111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f77be4f93cf401e8bafd3c2af25ab5f: Bootstrap complete.
I20260927 02:34:14.246572 9111 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f77be4f93cf401e8bafd3c2af25ab5f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f77be4f93cf401e8bafd3c2af25ab5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37517 } }
I20260927 02:34:14.246938 9111 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f77be4f93cf401e8bafd3c2af25ab5f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f77be4f93cf401e8bafd3c2af25ab5f, State: Initialized, Role: FOLLOWER
I20260927 02:34:14.247375 9111 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f77be4f93cf401e8bafd3c2af25ab5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f77be4f93cf401e8bafd3c2af25ab5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37517 } }
I20260927 02:34:14.247483 9111 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f77be4f93cf401e8bafd3c2af25ab5f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:34:14.247538 9111 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f77be4f93cf401e8bafd3c2af25ab5f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:34:14.247622 9111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f77be4f93cf401e8bafd3c2af25ab5f [term 1 FOLLOWER]: Advancing to term 2
I20260927 02:34:14.249470 9111 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f77be4f93cf401e8bafd3c2af25ab5f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f77be4f93cf401e8bafd3c2af25ab5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37517 } }
I20260927 02:34:14.249780 9111 leader_election.cc:302] T 00000000000000000000000000000000 P 5f77be4f93cf401e8bafd3c2af25ab5f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f77be4f93cf401e8bafd3c2af25ab5f; no voters:
I20260927 02:34:14.250062 9111 leader_election.cc:288] T 00000000000000000000000000000000 P 5f77be4f93cf401e8bafd3c2af25ab5f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 02:34:14.250152 9115 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f77be4f93cf401e8bafd3c2af25ab5f [term 2 FOLLOWER]: Leader election won for term 2
I20260927 02:34:14.250336 9115 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f77be4f93cf401e8bafd3c2af25ab5f [term 2 LEADER]: Becoming Leader. State: Replica: 5f77be4f93cf401e8bafd3c2af25ab5f, State: Running, Role: LEADER
I20260927 02:34:14.250692 9115 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f77be4f93cf401e8bafd3c2af25ab5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f77be4f93cf401e8bafd3c2af25ab5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37517 } }
I20260927 02:34:14.251225 9111 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f77be4f93cf401e8bafd3c2af25ab5f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:34:14.251545 9115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f77be4f93cf401e8bafd3c2af25ab5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f77be4f93cf401e8bafd3c2af25ab5f. Latest consensus state: current_term: 2 leader_uuid: "5f77be4f93cf401e8bafd3c2af25ab5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f77be4f93cf401e8bafd3c2af25ab5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37517 } } }
I20260927 02:34:14.251652 9115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f77be4f93cf401e8bafd3c2af25ab5f [sys.catalog]: This master's current role is: LEADER
I20260927 02:34:14.252224 9116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f77be4f93cf401e8bafd3c2af25ab5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5f77be4f93cf401e8bafd3c2af25ab5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f77be4f93cf401e8bafd3c2af25ab5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37517 } } }
I20260927 02:34:14.252317 9116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f77be4f93cf401e8bafd3c2af25ab5f [sys.catalog]: This master's current role is: LEADER
I20260927 02:34:14.252844 9119 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:34:14.256006 9119 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:34:14.256623 9119 catalog_manager.cc:1259] Loaded cluster ID: 617f15c67d844b3487e87a10737fb589
I20260927 02:34:14.256678 9119 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:34:14.258908 9119 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:34:14.259533 9119 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5f77be4f93cf401e8bafd3c2af25ab5f: Loaded TSK: 0
I20260927 02:34:14.260687 9119 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 02:34:14.331641 9113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:34:14.331874 9113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:34:14.331915 9113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:34:14.335911 9113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:34:14.335961 9113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:34:14.336023 9113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:34:14.341055 9113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45799
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44349
--tserver_master_addrs=127.0.98.62:37517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:34:14.342018 9113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:34:14.343050 9113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:34:14.352505 9137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:14.352507 9140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:14.352533 9138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:14.352707 9113 server_base.cc:1034] running on GCE node
I20260927 02:34:14.353295 9113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:34:14.353912 9113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:34:14.355087 9113 hybrid_clock.cc:648] HybridClock initialized: now 1790476454355070 us; error 32 us; skew 500 ppm
Sep 27 02:34:14 dist-test-slave-grlk krb5kdc[8530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476454, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:34:14.361927 9113 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 02:34:14.362890 9113 webserver.cc:460] Webserver started at http://127.0.98.1:44349/ using document root <none> and password file <none>
I20260927 02:34:14.363427 9113 fs_manager.cc:359] Metadata directory not provided
I20260927 02:34:14.363497 9113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:34:14.367256 9113 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260927 02:34:14.369220 9147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:14.370384 9113 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 02:34:14.370473 9113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-0/wal
uuid: "1f4a36d8def143e2aeb0d9ae9a1815ea"
format_stamp: "Formatted at 2026-09-27 02:34:13 on dist-test-slave-grlk"
server_key: "05aa757df543dad642d7117101dec484"
server_key_iv: "731e21ec09f1679ca041f6619c4fcfd0"
server_key_version: "encryptionkey@0"
I20260927 02:34:14.370980 9113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:34:14.386508 9113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:34:14.389391 9113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:34:14.389546 9113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:34:14.390049 9113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:34:14.390789 9113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:34:14.390831 9113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:14.390872 9113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:34:14.390900 9113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:14.398957 9113 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45799
I20260927 02:34:14.398981 9260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45799 every 8 connection(s)
I20260927 02:34:14.399850 9113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:34:14.408155 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 9113
I20260927 02:34:14.408603 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37045
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37517
--webserver_port=35037
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 02:34:14 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:14 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 02:34:14.411937 9264 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:14.401800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42911 (local address 127.0.98.62:37517)
0927 02:34:14.402208 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:14.402227 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:14.402383 (+ 156us) server_negotiation.cc:407] Connection header received
0927 02:34:14.403819 (+ 1436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:14.403830 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:14.404125 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:14.404385 (+ 260us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:14.405445 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:14.406388 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:14.407430 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:14.407864 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:14.407973 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:14.411329 (+ 3356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42911: BlockingRecv error: recv got EOF from 127.0.98.1:42911 (error 108)
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":100,"threads_started":1}
I20260927 02:34:14.411988 9263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:14.401790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.1:42911)
0927 02:34:14.402244 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:14.402285 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:14.403439 (+ 1154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:14.404495 (+ 1056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:14.404502 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:14.404829 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:14.405292 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:14.405305 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:14.406494 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:14.406497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:14.407334 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:14.407343 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:14.408000 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:14.408004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:14.408175 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:14.408740 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:14.408761 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:14.411144 (+ 2383us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":97,"threads_started":1}
W20260927 02:34:14.412834 9261 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37517 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:37517: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 02:34:14 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:14 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:14.417816 9263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:14.413302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.1:52729)
0927 02:34:14.413414 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:14.413428 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:14.413501 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:14.413736 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:14.413739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:14.413875 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:14.413959 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:14.413964 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:14.414659 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:14.414662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:14.415204 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:14.415212 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:14.415614 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:14.415618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:14.415706 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:14.416125 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:14.416136 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:14.417692 (+ 1556us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 02:34:14.417930 9264 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:14.413373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52729 (local address 127.0.98.62:37517)
0927 02:34:14.413469 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:14.413473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:14.413485 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:34:14.413592 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:14.413595 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:14.413635 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:14.413698 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:14.414091 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:14.414560 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:14.415315 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:14.415518 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:14.415602 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:14.417831 (+ 2229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52729: BlockingRecv error: recv got EOF from 127.0.98.1:52729 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 27 02:34:14 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:14 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:14.422703 9263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:14.418384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.1:59533)
0927 02:34:14.418491 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:14.418505 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:14.418575 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:14.418763 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:14.418765 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:14.418894 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:14.418958 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:14.418963 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:14.419636 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:14.419640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:14.420139 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:14.420146 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:14.420543 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:14.420546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:14.420615 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:14.421036 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:14.421048 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:14.422588 (+ 1540us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 02:34:14.422878 9264 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:14.418471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59533 (local address 127.0.98.62:37517)
0927 02:34:14.418580 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:14.418584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:14.418597 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:34:14.418636 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:14.418638 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:14.418676 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:14.418740 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:14.419060 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:14.419540 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:14.420244 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:14.420450 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:14.420541 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:14.422694 (+ 2153us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59533: BlockingRecv error: recv got EOF from 127.0.98.1:59533 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260927 02:34:14.422960 9261 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 02:34:14.516041 9265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:34:14.516310 9265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:34:14.516366 9265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:34:14.520367 9265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:34:14.520427 9265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:34:14.520509 9265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:34:14.525676 9265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37045
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35037
--tserver_master_addrs=127.0.98.62:37517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:34:14.526690 9265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:34:14.527642 9265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:34:14.536648 9271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:14.536734 9270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:14.536648 9273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:14.536999 9265 server_base.cc:1034] running on GCE node
I20260927 02:34:14.537389 9265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:34:14.537973 9265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:34:14.539134 9265 hybrid_clock.cc:648] HybridClock initialized: now 1790476454539102 us; error 44 us; skew 500 ppm
Sep 27 02:34:14 dist-test-slave-grlk krb5kdc[8530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476454, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:34:14.541720 9265 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 02:34:14.542496 9265 webserver.cc:460] Webserver started at http://127.0.98.2:35037/ using document root <none> and password file <none>
I20260927 02:34:14.542999 9265 fs_manager.cc:359] Metadata directory not provided
I20260927 02:34:14.543064 9265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:34:14.546725 9265 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260927 02:34:14.548640 9280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:14.549865 9265 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 02:34:14.549964 9265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-1/wal
uuid: "fc9e2a8a98fc4690a0de61167a3269d8"
format_stamp: "Formatted at 2026-09-27 02:34:13 on dist-test-slave-grlk"
server_key: "e3179fb6b8233898541c2cd4f6156fb7"
server_key_iv: "7ab194b862ea2e3cd2a4728ae055c51e"
server_key_version: "encryptionkey@0"
I20260927 02:34:14.550489 9265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:34:14.571398 9265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:34:14.574831 9265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:34:14.575088 9265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:34:14.575742 9265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:34:14.576581 9265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:34:14.576665 9265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:14.576721 9265 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:34:14.576752 9265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:14.585093 9265 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37045
I20260927 02:34:14.585119 9393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37045 every 8 connection(s)
I20260927 02:34:14.586072 9265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:34:14.595014 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 9265
Sep 27 02:34:14 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:14.595460 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43233
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37517
--webserver_port=37149
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 02:34:14 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 02:34:14.597182 9264 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:14.587923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51295 (local address 127.0.98.62:37517)
0927 02:34:14.588067 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:14.588072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:14.589138 (+ 1066us) server_negotiation.cc:407] Connection header received
0927 02:34:14.590442 (+ 1304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:14.590445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:14.590509 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:14.590587 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:14.591701 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:14.592228 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:14.593342 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:14.593579 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:14.593669 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:14.597060 (+ 3391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51295: BlockingRecv error: recv got EOF from 127.0.98.2:51295 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 02:34:14.597784 9396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:14.588360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.2:51295)
0927 02:34:14.588913 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:14.588992 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:14.590229 (+ 1237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:14.590779 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:14.590786 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:14.591116 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:14.591573 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:14.591585 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:14.592320 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:14.592322 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:14.593203 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:14.593211 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:14.593756 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:14.593761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:14.593914 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:14.594481 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:14.594498 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:14.596895 (+ 2397us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":140,"threads_started":1}
W20260927 02:34:14.598289 9394 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37517 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:37517: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 02:34:14 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:14 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:14.603456 9396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:14.598696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.2:53631)
0927 02:34:14.598849 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:14.598860 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:14.598918 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:14.599160 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:14.599162 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:14.599280 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:14.599368 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:14.599373 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:14.600326 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:14.600328 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:14.600847 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:14.600853 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:14.601396 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:14.601398 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:14.601463 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:14.601881 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:14.601891 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:14.603373 (+ 1482us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92}
W20260927 02:34:14.603639 9264 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:14.598775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53631 (local address 127.0.98.62:37517)
0927 02:34:14.598906 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:14.598911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:14.598927 (+ 16us) server_negotiation.cc:407] Connection header received
0927 02:34:14.599007 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:14.599011 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:14.599056 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:14.599132 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:14.599470 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:14.600232 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:14.600968 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:14.601300 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:14.601405 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:14.603524 (+ 2119us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53631: BlockingRecv error: recv got EOF from 127.0.98.2:53631 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 02:34:14 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:14 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:14.608795 9398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:14.603890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.2:32841)
0927 02:34:14.604169 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:14.604182 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:14.604261 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:14.604458 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:14.604460 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:14.604594 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:14.604691 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:14.604695 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:14.605663 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:14.605666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:14.606213 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:14.606220 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:14.606734 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:14.606737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:14.606804 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:14.607259 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:14.607269 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:14.608684 (+ 1415us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":96,"threads_started":1}
W20260927 02:34:14.608950 9264 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:14.603974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32841 (local address 127.0.98.62:37517)
0927 02:34:14.604084 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:14.604088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:14.604253 (+ 165us) server_negotiation.cc:407] Connection header received
0927 02:34:14.604323 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:14.604326 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:14.604369 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:14.604450 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:14.604789 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:14.605577 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:14.606311 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:14.606639 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:14.606730 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:14.608827 (+ 2097us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:32841: BlockingRecv error: recv got EOF from 127.0.98.2:32841 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260927 02:34:14.609427 9394 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 02:34:14.701568 9397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:34:14.701812 9397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:34:14.701853 9397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:34:14.705698 9397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:34:14.705746 9397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:34:14.705801 9397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:34:14.710764 9397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43233
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=37149
--tserver_master_addrs=127.0.98.62:37517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:34:14.711690 9397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:34:14.712560 9397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:34:14.721279 9406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:14.721334 9397 server_base.cc:1034] running on GCE node
W20260927 02:34:14.721299 9403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:14.721289 9404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:14.721844 9397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:34:14.722430 9397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:34:14.723577 9397 hybrid_clock.cc:648] HybridClock initialized: now 1790476454723553 us; error 40 us; skew 500 ppm
Sep 27 02:34:14 dist-test-slave-grlk krb5kdc[8530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476454, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:34:14.726143 9397 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 02:34:14.726892 9397 webserver.cc:460] Webserver started at http://127.0.98.3:37149/ using document root <none> and password file <none>
I20260927 02:34:14.727370 9397 fs_manager.cc:359] Metadata directory not provided
I20260927 02:34:14.727433 9397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:34:14.730908 9397 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:34:14.732688 9413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:14.733918 9397 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:34:14.734005 9397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-2/wal
uuid: "32354cfc52be47acb3d5408c78d79455"
format_stamp: "Formatted at 2026-09-27 02:34:13 on dist-test-slave-grlk"
server_key: "34a9aff041cc5146647359f927c44f3f"
server_key_iv: "ed7bdf7b1a1a4c187ae269442b128f9d"
server_key_version: "encryptionkey@0"
I20260927 02:34:14.734489 9397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:34:14.756646 9397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:34:14.759629 9397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:34:14.759768 9397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:34:14.760270 9397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:34:14.761034 9397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:34:14.761080 9397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:14.761125 9397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:34:14.761145 9397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:14.769407 9397 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43233
I20260927 02:34:14.769438 9526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43233 every 8 connection(s)
I20260927 02:34:14.770298 9397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:34:14.771170 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 9397
Sep 27 02:34:14 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:14 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:14.780400 9264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:14.771822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49206 (local address 127.0.98.62:37517)
0927 02:34:14.771950 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:14.771954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:14.772239 (+ 285us) server_negotiation.cc:407] Connection header received
0927 02:34:14.772369 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:14.772372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:14.772422 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:14.772506 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:14.773396 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:14.773988 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:14.775151 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:14.775356 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:14.775428 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:14.776678 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:14.776683 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:14.776694 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:14.776718 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:14.778468 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:14.778860 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:14.778866 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:14.778871 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:14.779015 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:14.779302 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:14.779305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:14.779306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:14.779642 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:14.779820 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:14.780069 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 02:34:14.780287 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 02:34:14.781061 9530 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:14.772206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46397 (local address 127.0.98.62:37517)
0927 02:34:14.772688 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:14.772692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:14.773066 (+ 374us) server_negotiation.cc:407] Connection header received
0927 02:34:14.774592 (+ 1526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:14.774594 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:14.774631 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:14.774680 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:14.775977 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:14.776468 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:14.777601 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:14.777809 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:14.777882 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:14.780967 (+ 3085us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46397: BlockingRecv error: recv got EOF from 127.0.98.3:46397 (error 108)
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":78,"threads_started":1}
I20260927 02:34:14.781664 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:14.772383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.3:46397)
0927 02:34:14.772899 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:14.772972 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:14.774390 (+ 1418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:14.774840 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:14.774848 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:14.775274 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:14.775842 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:14.775858 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:14.776592 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:14.776596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:14.777500 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:14.777509 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:14.777945 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:14.777949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:14.778123 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:14.778576 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:14.778594 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:14.780858 (+ 2264us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":116,"threads_started":1}
W20260927 02:34:14.782034 9527 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37517 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:37517: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 02:34:14 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:14 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:14.786377 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:14.782417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.3:44531)
0927 02:34:14.782497 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:14.782509 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:14.782570 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:14.782816 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:14.782818 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:14.782961 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:14.783049 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:14.783053 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:14.783646 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:14.783648 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:14.784157 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:14.784162 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:14.784530 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:14.784533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:14.784591 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:14.784894 (+ 303us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:14.784903 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:14.786299 (+ 1396us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":23}
W20260927 02:34:14.786506 9530 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:14.782474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44531 (local address 127.0.98.62:37517)
0927 02:34:14.782667 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:14.782669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:14.782677 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:34:14.782697 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:14.782699 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:14.782726 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:14.782778 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:14.783125 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:14.783574 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:14.784258 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:14.784454 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:14.784526 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:14.786403 (+ 1877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44531: BlockingRecv error: recv got EOF from 127.0.98.3:44531 (error 108)
Metrics: {"server-negotiator.queue_time_us":143}
Sep 27 02:34:14 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:14 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:14.790670 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:14.786764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.3:47595)
0927 02:34:14.786885 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:14.786899 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:14.786961 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:14.787117 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:14.787119 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:14.787223 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:14.787283 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:14.787287 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:14.787907 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:14.787910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:14.788397 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:14.788404 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:14.788804 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:14.788807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:14.788872 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:14.789241 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:14.789252 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:14.790569 (+ 1317us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 02:34:14.790771 9530 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:14.786828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47595 (local address 127.0.98.62:37517)
0927 02:34:14.786949 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:14.786952 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:14.786963 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:34:14.787012 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:14.787014 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:14.787042 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:14.787096 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:14.787355 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:14.787797 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:14.788486 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:14.788711 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:14.788798 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:14.790686 (+ 1888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47595: BlockingRecv error: recv got EOF from 127.0.98.3:47595 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260927 02:34:14.790851 9527 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 02:34:15 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:15 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:15.429157 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:15.423662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.1:55777)
0927 02:34:15.424006 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:15.424023 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:15.424134 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:15.424403 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:15.424406 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:15.424631 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:15.424748 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:15.424755 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:15.425590 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:15.425594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:15.426170 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:15.426178 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:15.426610 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:15.426614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:15.426691 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:15.427227 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:15.427238 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:15.428998 (+ 1760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":97,"threads_started":1}
W20260927 02:34:15.429256 9533 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:15.423816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55777 (local address 127.0.98.62:37517)
0927 02:34:15.424121 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:15.424126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:15.424140 (+ 14us) server_negotiation.cc:407] Connection header received
0927 02:34:15.424238 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:15.424241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:15.424287 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:15.424360 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:15.424864 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:15.425482 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:15.426292 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:15.426511 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:15.426593 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:15.429140 (+ 2547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55777: BlockingRecv error: recv got EOF from 127.0.98.1:55777 (error 108)
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":172,"threads_started":1}
Sep 27 02:34:15 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:15 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:15.615020 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:15.609997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.2:45059)
0927 02:34:15.610271 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:15.610285 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:15.610383 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:15.610628 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:15.610630 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:15.610810 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:15.610929 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:15.610934 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:15.611635 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:15.611638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:15.612216 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:15.612224 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:15.612628 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:15.612631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:15.612712 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:15.613252 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:15.613262 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:15.614894 (+ 1632us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":86,"threads_started":1}
W20260927 02:34:15.615113 9533 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:15.610068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45059 (local address 127.0.98.62:37517)
0927 02:34:15.610194 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:15.610198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:15.610346 (+ 148us) server_negotiation.cc:407] Connection header received
0927 02:34:15.610461 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:15.610463 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:15.610510 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:15.610598 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:15.611014 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:15.611527 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:15.612301 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:15.612501 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:15.612565 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:15.615017 (+ 2452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45059: BlockingRecv error: recv got EOF from 127.0.98.2:45059 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 27 02:34:15 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:15 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:15.796324 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:15.791382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.3:49519)
0927 02:34:15.791659 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:15.791672 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:15.791764 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:15.791968 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:15.791970 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:15.792143 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:15.792258 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:15.792263 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:15.792920 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:15.792922 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:15.793513 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:15.793520 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:15.793889 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:15.793892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:15.793956 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:15.794458 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:15.794470 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:15.796198 (+ 1728us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":80,"threads_started":1}
W20260927 02:34:15.796429 9533 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:15.791438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49519 (local address 127.0.98.62:37517)
0927 02:34:15.791550 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:15.791553 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:15.791730 (+ 177us) server_negotiation.cc:407] Connection header received
0927 02:34:15.791832 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:15.791834 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:15.791875 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:15.791939 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:15.792342 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:15.792843 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:15.793595 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:15.793809 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:15.793876 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:15.796330 (+ 2454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49519: BlockingRecv error: recv got EOF from 127.0.98.3:49519 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 27 02:34:16 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:16 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:16.435012 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:16.429884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.1:45301)
0927 02:34:16.430162 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:16.430175 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:16.430269 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:16.430565 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:16.430568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:16.430745 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:16.430844 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:16.430849 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:16.431577 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:16.431580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:16.432154 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:16.432163 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:16.432535 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:16.432538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:16.432595 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:16.433175 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:16.433187 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:16.434879 (+ 1692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":70,"threads_started":1}
W20260927 02:34:16.435123 9537 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:16.429962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45301 (local address 127.0.98.62:37517)
0927 02:34:16.430183 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:16.430194 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:16.430221 (+ 27us) server_negotiation.cc:407] Connection header received
0927 02:34:16.430354 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:16.430358 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:16.430408 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:16.430488 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:16.430934 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:16.431474 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:16.432243 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:16.432455 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:16.432521 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:16.435009 (+ 2488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45301: BlockingRecv error: recv got EOF from 127.0.98.1:45301 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":63,"threads_started":1}
Sep 27 02:34:16 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:16 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:16.620671 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:16.615741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.2:59081)
0927 02:34:16.615996 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:16.616009 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:16.616106 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:16.616308 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:16.616310 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:16.616478 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:16.616571 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:16.616576 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:16.617335 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:16.617338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:16.617901 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:16.617910 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:16.618277 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:16.618280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:16.618337 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:16.618813 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:16.618824 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:16.620491 (+ 1667us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":73,"threads_started":1}
W20260927 02:34:16.620714 9537 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:16.615836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59081 (local address 127.0.98.62:37517)
0927 02:34:16.615970 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:16.615973 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:16.616061 (+ 88us) server_negotiation.cc:407] Connection header received
0927 02:34:16.616180 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:16.616182 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:16.616224 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:16.616278 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:16.616665 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:16.617240 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:16.617987 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:16.618201 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:16.618267 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:16.620622 (+ 2355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59081: BlockingRecv error: recv got EOF from 127.0.98.2:59081 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 27 02:34:16 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:16 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:16.802042 9539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:16.797107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.3:36887)
0927 02:34:16.797348 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:16.797363 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:16.797469 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:16.797720 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:16.797723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:16.797903 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:16.797991 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:16.797997 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:16.798706 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:16.798710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:16.799260 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:16.799268 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:16.799674 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:16.799678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:16.799757 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:16.800232 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:16.800244 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:16.801900 (+ 1656us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":65,"threads_started":1}
W20260927 02:34:16.802129 9537 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:16.797272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36887 (local address 127.0.98.62:37517)
0927 02:34:16.797414 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:16.797418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:16.797430 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:34:16.797557 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:16.797572 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:16.797614 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:16.797677 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:16.798099 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:16.798604 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:16.799371 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:16.799576 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:16.799660 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:16.802032 (+ 2372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36887: BlockingRecv error: recv got EOF from 127.0.98.3:36887 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 02:34:17 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:17 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:17.441156 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:17.435737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.1:58697)
0927 02:34:17.436046 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:17.436061 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:17.436167 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:17.436390 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:17.436392 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:17.436577 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:17.436676 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:17.436682 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:17.437537 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:17.437539 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:17.438125 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:17.438137 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:17.438559 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:17.438562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:17.438624 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:17.439066 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:17.439077 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:17.440953 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":82,"threads_started":1}
W20260927 02:34:17.441212 9541 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:17.435850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58697 (local address 127.0.98.62:37517)
0927 02:34:17.436051 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:17.436055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:17.436129 (+ 74us) server_negotiation.cc:407] Connection header received
0927 02:34:17.436251 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:17.436253 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:17.436293 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:17.436359 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:17.436764 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:17.437438 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:17.438255 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:17.438470 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:17.438544 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:17.441084 (+ 2540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58697: BlockingRecv error: recv got EOF from 127.0.98.1:58697 (error 108)
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":52,"threads_started":1}
Sep 27 02:34:17 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:17 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:17.634316 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:17.621447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.2:50369)
0927 02:34:17.621690 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:17.621702 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:17.621800 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:17.621987 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:17.621989 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:17.622175 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:17.622267 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:17.622272 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:17.622931 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:17.622934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:17.623475 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:17.623481 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:17.623823 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:17.623825 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:17.623881 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:17.632072 (+ 8191us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:17.632085 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:17.634189 (+ 2104us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":76,"threads_started":1}
W20260927 02:34:17.634455 9541 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:17.621529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50369 (local address 127.0.98.62:37517)
0927 02:34:17.621668 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:17.621671 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:17.621765 (+ 94us) server_negotiation.cc:407] Connection header received
0927 02:34:17.621869 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:17.621871 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:17.621911 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:17.621976 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:17.622352 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:17.622856 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:17.623551 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:17.623749 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:17.623812 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:17.634338 (+ 10526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50369: BlockingRecv error: recv got EOF from 127.0.98.2:50369 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 02:34:17 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:17 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:17.809103 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:17.802757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.3:50837)
0927 02:34:17.802988 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:17.803002 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:17.803097 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:17.803355 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:17.803358 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:17.803570 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:17.803689 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:17.803696 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:17.804705 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:17.804708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:17.805612 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:17.805622 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:17.806205 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:17.806209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:17.806292 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:17.806834 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:17.806848 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:17.808907 (+ 2059us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":67,"threads_started":1}
W20260927 02:34:17.809525 9541 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:17.802838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50837 (local address 127.0.98.62:37517)
0927 02:34:17.802972 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:17.802976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:17.803055 (+ 79us) server_negotiation.cc:407] Connection header received
0927 02:34:17.803165 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:17.803168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:17.803224 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:17.803315 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:17.803785 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:17.804606 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:17.805733 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:17.806082 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:17.806188 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:17.809429 (+ 3241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50837: BlockingRecv error: recv got EOF from 127.0.98.3:50837 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 02:34:18 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:18 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:18.446674 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:18.441884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.1:42957)
0927 02:34:18.442120 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:18.442133 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:18.442216 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:18.442407 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:18.442409 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:18.442583 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:18.442676 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:18.442681 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:18.443413 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:18.443415 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:18.443954 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:18.443961 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:18.444324 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:18.444326 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:18.444384 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:18.444830 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:18.444841 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:18.446565 (+ 1724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":80,"threads_started":1}
W20260927 02:34:18.446851 9545 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:18.441943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42957 (local address 127.0.98.62:37517)
0927 02:34:18.442150 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:18.442153 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:18.442170 (+ 17us) server_negotiation.cc:407] Connection header received
0927 02:34:18.442287 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:18.442288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:18.442327 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:18.442397 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:18.442758 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:18.443332 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:18.444033 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:18.444246 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:18.444309 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:18.446718 (+ 2409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42957: BlockingRecv error: recv got EOF from 127.0.98.1:42957 (error 108)
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":68,"threads_started":1}
Sep 27 02:34:18 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:18 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:18.640337 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:18.635047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.2:34305)
0927 02:34:18.635293 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:18.635306 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:18.635399 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:18.635617 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:18.635619 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:18.635788 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:18.635881 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:18.635886 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:18.636618 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:18.636621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:18.637365 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:18.637377 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:18.637802 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:18.637805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:18.637877 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:18.638384 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:18.638397 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:18.640194 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":72,"threads_started":1}
W20260927 02:34:18.640415 9545 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:18.635145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34305 (local address 127.0.98.62:37517)
0927 02:34:18.635307 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:18.635310 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:18.635355 (+ 45us) server_negotiation.cc:407] Connection header received
0927 02:34:18.635477 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:18.635480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:18.635522 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:18.635591 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:18.635974 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:18.636497 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:18.637470 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:18.637688 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:18.637761 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:18.640321 (+ 2560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34305: BlockingRecv error: recv got EOF from 127.0.98.2:34305 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 27 02:34:18 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:18 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:18.814996 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:18.809883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.3:53949)
0927 02:34:18.810152 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:18.810165 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:18.810282 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:18.810520 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:18.810522 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:18.810703 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:18.810792 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:18.810797 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:18.811517 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:18.811520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:18.812058 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:18.812064 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:18.812488 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:18.812492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:18.812570 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:18.813076 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:18.813088 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:18.814858 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":95,"threads_started":1}
W20260927 02:34:18.815091 9545 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:18.809978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53949 (local address 127.0.98.62:37517)
0927 02:34:18.810117 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:18.810122 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:18.810230 (+ 108us) server_negotiation.cc:407] Connection header received
0927 02:34:18.810369 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:18.810372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:18.810421 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:18.810484 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:18.810901 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:18.811417 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:18.812168 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:18.812390 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:18.812474 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:18.814994 (+ 2520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53949: BlockingRecv error: recv got EOF from 127.0.98.3:53949 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 27 02:34:19 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:19 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:19.459558 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:19.447378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.1:45461)
0927 02:34:19.447689 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:19.447703 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:19.447799 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:19.448039 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:19.448041 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:19.448213 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:19.448307 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:19.448313 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:19.449050 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:19.449053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:19.449905 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:19.449920 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:19.450654 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:19.450657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:19.450723 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:19.457665 (+ 6942us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:19.457678 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:19.459444 (+ 1766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":102,"threads_started":1}
W20260927 02:34:19.459733 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:19.447478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45461 (local address 127.0.98.62:37517)
0927 02:34:19.447717 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:19.447721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:19.447757 (+ 36us) server_negotiation.cc:407] Connection header received
0927 02:34:19.447884 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:19.447887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:19.447935 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:19.448008 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:19.448393 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:19.448901 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:19.450041 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:19.450524 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:19.450612 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:19.459604 (+ 8992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45461: BlockingRecv error: recv got EOF from 127.0.98.1:45461 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":73,"threads_started":1}
Sep 27 02:34:19 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:19 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:19.646174 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:19.641239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.2:40963)
0927 02:34:19.641488 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:19.641501 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:19.641602 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:19.641858 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:19.641862 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:19.642082 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:19.642189 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:19.642195 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:19.642873 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:19.642875 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:19.643456 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:19.643463 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:19.643858 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:19.643860 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:19.643915 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:19.644393 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:19.644407 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:19.646044 (+ 1637us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":73,"threads_started":1}
W20260927 02:34:19.646265 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:19.641382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40963 (local address 127.0.98.62:37517)
0927 02:34:19.641504 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:19.641508 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:19.641585 (+ 77us) server_negotiation.cc:407] Connection header received
0927 02:34:19.641689 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:19.641692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:19.641743 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:19.641816 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:19.642287 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:19.642798 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:19.643552 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:19.643779 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:19.643853 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:19.646160 (+ 2307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40963: BlockingRecv error: recv got EOF from 127.0.98.2:40963 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 27 02:34:19 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:19 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:19.820792 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:19.815738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.3:35651)
0927 02:34:19.815994 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:19.816007 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:19.816104 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:19.816336 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:19.816338 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:19.816526 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:19.816616 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:19.816621 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:19.817413 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:19.817416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:19.817990 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:19.817998 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:19.818440 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:19.818443 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:19.818519 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:19.819001 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:19.819013 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:19.820632 (+ 1619us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":79,"threads_started":1}
W20260927 02:34:19.820874 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:19.815808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35651 (local address 127.0.98.62:37517)
0927 02:34:19.815948 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:19.815953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:19.816070 (+ 117us) server_negotiation.cc:407] Connection header received
0927 02:34:19.816188 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:19.816191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:19.816239 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:19.816303 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:19.816731 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:19.817314 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:19.818097 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:19.818333 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:19.818424 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:19.820774 (+ 2350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35651: BlockingRecv error: recv got EOF from 127.0.98.3:35651 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 02:34:20 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:20 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:20.465524 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:20.460296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.1:52835)
0927 02:34:20.460524 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:20.460537 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:20.460621 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:20.460843 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:20.460845 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:20.461039 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:20.461127 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:20.461132 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:20.461873 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:20.461876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:20.462415 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:20.462423 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:20.462854 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:20.462856 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:20.462910 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:20.463384 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:20.463395 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:20.465377 (+ 1982us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":66,"threads_started":1}
W20260927 02:34:20.465701 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:20.460348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52835 (local address 127.0.98.62:37517)
0927 02:34:20.460576 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:20.460579 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:20.460593 (+ 14us) server_negotiation.cc:407] Connection header received
0927 02:34:20.460694 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:20.460696 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:20.460736 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:20.460811 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:20.461228 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:20.461773 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:20.462531 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:20.462766 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:20.462848 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:20.465567 (+ 2719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52835: BlockingRecv error: recv got EOF from 127.0.98.1:52835 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":66,"threads_started":1}
Sep 27 02:34:20 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:20 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:20.652037 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:20.646832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.2:53597)
0927 02:34:20.647111 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:20.647125 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:20.647231 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:20.647528 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:20.647531 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:20.647708 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:20.647809 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:20.647815 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:20.648560 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:20.648564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:20.649185 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:20.649195 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:20.649671 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:20.649673 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:20.649742 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:20.650210 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:20.650221 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:20.651924 (+ 1703us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":82,"threads_started":1}
W20260927 02:34:20.652160 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:20.646912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53597 (local address 127.0.98.62:37517)
0927 02:34:20.647041 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:20.647044 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:20.647189 (+ 145us) server_negotiation.cc:407] Connection header received
0927 02:34:20.647344 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:20.647347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:20.647393 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:20.647465 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:20.647934 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:20.648445 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:20.649282 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:20.649573 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:20.649658 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:20.652044 (+ 2386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53597: BlockingRecv error: recv got EOF from 127.0.98.2:53597 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 02:34:20 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:20 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:20.826365 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:20.821698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.3:38623)
0927 02:34:20.821940 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:20.821952 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:20.822053 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:20.822253 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:20.822255 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:20.822417 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:20.822509 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:20.822514 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:20.823167 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:20.823169 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:20.823717 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:20.823723 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:20.824066 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:20.824068 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:20.824121 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:20.824623 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:20.824635 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:20.826245 (+ 1610us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":67,"threads_started":1}
W20260927 02:34:20.826459 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:20.821793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38623 (local address 127.0.98.62:37517)
0927 02:34:20.821932 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:20.821935 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:20.822011 (+ 76us) server_negotiation.cc:407] Connection header received
0927 02:34:20.822120 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:20.822122 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:20.822165 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:20.822228 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:20.822585 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:20.823095 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:20.823794 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:20.823995 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:20.824056 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:20.826362 (+ 2306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38623: BlockingRecv error: recv got EOF from 127.0.98.3:38623 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 27 02:34:21 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:21 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:21.471931 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:21.466601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.1:32857)
0927 02:34:21.466921 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:21.466935 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:21.467038 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:21.467270 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:21.467272 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:21.467555 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:21.467654 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:21.467661 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:21.468413 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:21.468415 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:21.469002 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:21.469010 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:21.469463 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:21.469467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:21.469550 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:21.470063 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:21.470077 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:21.471784 (+ 1707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":97,"threads_started":1}
W20260927 02:34:21.472036 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:21.466692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32857 (local address 127.0.98.62:37517)
0927 02:34:21.466939 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:21.466944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:21.466995 (+ 51us) server_negotiation.cc:407] Connection header received
0927 02:34:21.467120 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:21.467122 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:21.467164 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:21.467241 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:21.467777 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:21.468317 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:21.469107 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:21.469349 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:21.469434 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:21.471921 (+ 2487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:32857: BlockingRecv error: recv got EOF from 127.0.98.1:32857 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":73,"threads_started":1}
Sep 27 02:34:21 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:21 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:21.657665 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:21.652704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.2:41193)
0927 02:34:21.653033 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:21.653046 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:21.653138 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:21.653388 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:21.653391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:21.653556 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:21.653647 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:21.653652 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:21.654324 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:21.654327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:21.654908 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:21.654916 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:21.655326 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:21.655330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:21.655411 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:21.655869 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:21.655880 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:21.657525 (+ 1645us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":90,"threads_started":1}
W20260927 02:34:21.657754 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:21.652736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41193 (local address 127.0.98.62:37517)
0927 02:34:21.652882 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:21.652885 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:21.653105 (+ 220us) server_negotiation.cc:407] Connection header received
0927 02:34:21.653224 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:21.653227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:21.653280 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:21.653346 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:21.653732 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:21.654242 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:21.654992 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:21.655208 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:21.655278 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:21.657646 (+ 2368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41193: BlockingRecv error: recv got EOF from 127.0.98.2:41193 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 27 02:34:21 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:21 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:21.831983 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:21.827091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.3:54207)
0927 02:34:21.827406 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:21.827418 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:21.827521 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:21.827764 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:21.827767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:21.827943 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:21.828030 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:21.828035 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:21.828751 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:21.828754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:21.829316 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:21.829324 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:21.829729 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:21.829732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:21.829808 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:21.830262 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:21.830274 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:21.831874 (+ 1600us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":99,"threads_started":1}
W20260927 02:34:21.832096 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:21.827156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54207 (local address 127.0.98.62:37517)
0927 02:34:21.827308 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:21.827313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:21.827487 (+ 174us) server_negotiation.cc:407] Connection header received
0927 02:34:21.827605 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:21.827608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:21.827655 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:21.827717 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:21.828134 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:21.828648 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:21.829419 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:21.829632 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:21.829713 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:21.831987 (+ 2274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54207: BlockingRecv error: recv got EOF from 127.0.98.3:54207 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 27 02:34:22 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:22 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:22.477710 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:22.472641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.1:39451)
0927 02:34:22.472869 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:22.472881 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:22.472990 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:22.473194 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:22.473196 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:22.473363 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:22.473455 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:22.473460 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:22.474208 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:22.474211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:22.474806 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:22.474817 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:22.475235 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:22.475239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:22.475308 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:22.475739 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:22.475749 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:22.477566 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":62,"threads_started":1}
W20260927 02:34:22.477788 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:22.472754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39451 (local address 127.0.98.62:37517)
0927 02:34:22.472986 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:22.472991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:22.473002 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:34:22.473054 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:22.473056 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:22.473095 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:22.473161 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:22.473537 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:22.474096 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:22.474909 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:22.475127 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:22.475198 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:22.477678 (+ 2480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39451: BlockingRecv error: recv got EOF from 127.0.98.1:39451 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":68,"threads_started":1}
Sep 27 02:34:22 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:22 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:22.663321 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:22.658367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.2:33453)
0927 02:34:22.658591 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:22.658604 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:22.658700 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:22.658934 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:22.658937 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:22.659095 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:22.659188 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:22.659193 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:22.660036 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:22.660040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:22.660623 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:22.660631 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:22.661051 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:22.661054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:22.661114 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:22.661543 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:22.661557 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:22.663178 (+ 1621us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":68,"threads_started":1}
W20260927 02:34:22.663414 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:22.658408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33453 (local address 127.0.98.62:37517)
0927 02:34:22.658534 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:22.658537 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:22.658663 (+ 126us) server_negotiation.cc:407] Connection header received
0927 02:34:22.658765 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:22.658767 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:22.658810 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:22.658872 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:22.659283 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:22.659919 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:22.660715 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:22.660915 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:22.661024 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:22.663319 (+ 2295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33453: BlockingRecv error: recv got EOF from 127.0.98.2:33453 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 27 02:34:22 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:22 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:22.837486 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:22.832732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.3:59327)
0927 02:34:22.832992 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:22.833005 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:22.833116 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:22.833313 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:22.833314 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:22.833476 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:22.833564 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:22.833569 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:22.834236 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:22.834237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:22.834779 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:22.834785 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:22.835169 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:22.835172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:22.835232 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:22.835654 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:22.835664 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:22.837374 (+ 1710us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":70,"threads_started":1}
W20260927 02:34:22.837584 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:22.832791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59327 (local address 127.0.98.62:37517)
0927 02:34:22.832924 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:22.832949 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:22.833062 (+ 113us) server_negotiation.cc:407] Connection header received
0927 02:34:22.833199 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:22.833201 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:22.833245 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:22.833301 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:22.833641 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:22.834156 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:22.834870 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:22.835078 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:22.835149 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:22.837488 (+ 2339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59327: BlockingRecv error: recv got EOF from 127.0.98.3:59327 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 27 02:34:23 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:23 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:23.483156 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:23.478342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.1:42651)
0927 02:34:23.478592 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:23.478605 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:23.478689 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:23.478900 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:23.478902 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:23.479067 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:23.479160 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:23.479165 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:23.479841 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:23.479843 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:23.480396 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:23.480402 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:23.480777 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:23.480779 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:23.480832 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:23.481283 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:23.481294 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:23.483012 (+ 1718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":85,"threads_started":1}
W20260927 02:34:23.483258 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:23.478428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42651 (local address 127.0.98.62:37517)
0927 02:34:23.478658 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:23.478661 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:23.478673 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:34:23.478763 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:23.478766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:23.478806 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:23.478879 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:23.479254 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:23.479769 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:23.480487 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:23.480700 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:23.480772 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:23.483146 (+ 2374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42651: BlockingRecv error: recv got EOF from 127.0.98.1:42651 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":73,"threads_started":1}
Sep 27 02:34:23 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:23 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:23.669093 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:23.664091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.2:34443)
0927 02:34:23.664385 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:23.664396 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:23.664503 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:23.664745 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:23.664748 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:23.664961 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:23.665065 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:23.665072 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:23.665781 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:23.665785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:23.666340 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:23.666347 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:23.666748 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:23.666751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:23.666826 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:23.667285 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:23.667300 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:23.668955 (+ 1655us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":68,"threads_started":1}
W20260927 02:34:23.669184 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:23.664131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34443 (local address 127.0.98.62:37517)
0927 02:34:23.664272 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:23.664276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:23.664460 (+ 184us) server_negotiation.cc:407] Connection header received
0927 02:34:23.664588 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:23.664591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:23.664635 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:23.664701 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:23.665165 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:23.665683 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:23.666443 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:23.666655 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:23.666736 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:23.669079 (+ 2343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34443: BlockingRecv error: recv got EOF from 127.0.98.2:34443 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 27 02:34:23 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:23 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:23.842962 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:23.838201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.3:42721)
0927 02:34:23.838455 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:23.838467 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:23.838559 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:23.838754 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:23.838756 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:23.838924 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:23.839019 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:23.839024 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:23.839696 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:23.839698 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:23.840258 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:23.840265 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:23.840680 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:23.840683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:23.840744 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:23.841219 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:23.841230 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:23.842848 (+ 1618us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":81,"threads_started":1}
W20260927 02:34:23.843097 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:23.838287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42721 (local address 127.0.98.62:37517)
0927 02:34:23.838407 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:23.838412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:23.838517 (+ 105us) server_negotiation.cc:407] Connection header received
0927 02:34:23.838622 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:23.838624 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:23.838666 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:23.838728 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:23.839096 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:23.839607 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:23.840360 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:23.840581 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:23.840656 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:23.842988 (+ 2332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42721: BlockingRecv error: recv got EOF from 127.0.98.3:42721 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 27 02:34:24 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:24 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:24.491570 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:24.483910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.1:45187)
0927 02:34:24.484202 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:24.484215 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:24.484308 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:24.484528 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:24.484530 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:24.484697 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:24.484794 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:24.484800 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:24.485587 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:24.485590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:24.486145 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:24.486153 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:24.486590 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:24.486592 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:24.486648 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:24.489853 (+ 3205us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:24.489865 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:24.491459 (+ 1594us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":106,"threads_started":1}
W20260927 02:34:24.491750 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:24.483974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45187 (local address 127.0.98.62:37517)
0927 02:34:24.484220 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:24.484223 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:24.484284 (+ 61us) server_negotiation.cc:407] Connection header received
0927 02:34:24.484381 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:24.484383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:24.484426 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:24.484517 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:24.484880 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:24.485451 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:24.486279 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:24.486502 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:24.486587 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:24.491602 (+ 5015us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45187: BlockingRecv error: recv got EOF from 127.0.98.1:45187 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":75,"threads_started":1}
Sep 27 02:34:24 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:24 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:24.674646 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:24.669793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.2:59273)
0927 02:34:24.670016 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:24.670028 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:24.670117 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:24.670336 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:24.670338 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:24.670503 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:24.670597 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:24.670602 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:24.671258 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:24.671260 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:24.671822 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:24.671832 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:24.672338 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:24.672341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:24.672425 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:24.672922 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:24.672948 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:24.674538 (+ 1590us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":64,"threads_started":1}
W20260927 02:34:24.674753 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:24.669865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59273 (local address 127.0.98.62:37517)
0927 02:34:24.670000 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:24.670003 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:24.670086 (+ 83us) server_negotiation.cc:407] Connection header received
0927 02:34:24.670209 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:24.670211 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:24.670253 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:24.670312 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:24.670674 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:24.671171 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:24.671922 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:24.672225 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:24.672335 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:24.674649 (+ 2314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59273: BlockingRecv error: recv got EOF from 127.0.98.2:59273 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 02:34:24 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:24 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:24.849069 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:24.843677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.3:39759)
0927 02:34:24.843958 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:24.843973 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:24.844072 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:24.844306 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:24.844309 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:24.844497 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:24.844588 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:24.844594 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:24.845360 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:24.845363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:24.846079 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:24.846090 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:24.846535 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:24.846539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:24.846614 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:24.847116 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:24.847127 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:24.848912 (+ 1785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":93,"threads_started":1}
W20260927 02:34:24.849201 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:24.843821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39759 (local address 127.0.98.62:37517)
0927 02:34:24.843958 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:24.843962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:24.844025 (+ 63us) server_negotiation.cc:407] Connection header received
0927 02:34:24.844154 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:24.844157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:24.844208 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:24.844270 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:24.844693 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:24.845260 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:24.846213 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:24.846435 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:24.846520 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:24.849093 (+ 2573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39759: BlockingRecv error: recv got EOF from 127.0.98.3:39759 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 27 02:34:25 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:25 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:25.497169 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:25.492297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.1:47195)
0927 02:34:25.492562 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:25.492575 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:25.492673 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:25.492949 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:25.492951 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:25.493123 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:25.493215 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:25.493220 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:25.493930 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:25.493932 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:25.494483 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:25.494490 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:25.494878 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:25.494880 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:25.494937 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:25.495393 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:25.495404 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:25.497037 (+ 1633us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":84,"threads_started":1}
W20260927 02:34:25.497270 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:25.492401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47195 (local address 127.0.98.62:37517)
0927 02:34:25.492627 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:25.492630 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:25.492643 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:34:25.492783 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:25.492786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:25.492833 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:25.492911 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:25.493299 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:25.493838 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:25.494566 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:25.494799 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:25.494871 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:25.497149 (+ 2278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47195: BlockingRecv error: recv got EOF from 127.0.98.1:47195 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":69,"threads_started":1}
Sep 27 02:34:25 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:25 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:25.680275 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:25.675271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.2:44063)
0927 02:34:25.675543 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:25.675556 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:25.675658 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:25.675901 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:25.675903 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:25.676096 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:25.676200 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:25.676206 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:25.676913 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:25.676917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:25.677513 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:25.677521 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:25.677934 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:25.677938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:25.678017 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:25.678446 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:25.678457 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:25.680162 (+ 1705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":91,"threads_started":1}
W20260927 02:34:25.680421 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:25.675342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44063 (local address 127.0.98.62:37517)
0927 02:34:25.675475 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:25.675479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:25.675620 (+ 141us) server_negotiation.cc:407] Connection header received
0927 02:34:25.675746 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:25.675749 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:25.675792 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:25.675866 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:25.676307 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:25.676816 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:25.677623 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:25.677840 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:25.677920 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:25.680307 (+ 2387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44063: BlockingRecv error: recv got EOF from 127.0.98.2:44063 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 27 02:34:25 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:25 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:25.855108 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:25.849801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.3:40853)
0927 02:34:25.850107 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:25.850136 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:25.850246 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:25.850483 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:25.850486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:25.850718 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:25.850829 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:25.850836 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:25.851745 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:25.851748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:25.852365 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:25.852375 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:25.852747 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:25.852750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:25.852807 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:25.853268 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:25.853278 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:25.854994 (+ 1716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":97,"threads_started":1}
W20260927 02:34:25.855235 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:25.849854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40853 (local address 127.0.98.62:37517)
0927 02:34:25.850031 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:25.850035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:25.850206 (+ 171us) server_negotiation.cc:407] Connection header received
0927 02:34:25.850335 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:25.850338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:25.850386 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:25.850458 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:25.850941 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:25.851657 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:25.852451 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:25.852667 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:25.852740 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:25.855106 (+ 2366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40853: BlockingRecv error: recv got EOF from 127.0.98.3:40853 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 27 02:34:26 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:26 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:26.503849 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:26.497920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.1:52559)
0927 02:34:26.498204 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:26.498221 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:26.498328 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:26.498586 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:26.498588 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:26.498806 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:26.498928 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:26.498935 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:26.499672 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:26.499675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:26.500265 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:26.500273 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:26.500777 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:26.500781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:26.500871 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:26.501564 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:26.501578 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:26.503699 (+ 2121us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":69,"threads_started":1}
W20260927 02:34:26.503954 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:26.497988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52559 (local address 127.0.98.62:37517)
0927 02:34:26.498211 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:26.498216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:26.498287 (+ 71us) server_negotiation.cc:407] Connection header received
0927 02:34:26.498405 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:26.498408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:26.498459 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:26.498546 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:26.499025 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:26.499580 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:26.500384 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:26.500648 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:26.500754 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:26.503840 (+ 3086us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52559: BlockingRecv error: recv got EOF from 127.0.98.1:52559 (error 108)
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":64,"threads_started":1}
Sep 27 02:34:26 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:26 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:26.685967 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:26.681003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.2:35181)
0927 02:34:26.681232 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:26.681244 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:26.681340 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:26.681599 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:26.681601 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:26.681774 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:26.681869 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:26.681874 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:26.682635 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:26.682638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:26.683200 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:26.683208 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:26.683572 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:26.683575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:26.683631 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:26.684081 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:26.684091 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:26.685799 (+ 1708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":77,"threads_started":1}
W20260927 02:34:26.686036 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:26.681037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35181 (local address 127.0.98.62:37517)
0927 02:34:26.681214 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:26.681219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:26.681303 (+ 84us) server_negotiation.cc:407] Connection header received
0927 02:34:26.681426 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:26.681429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:26.681476 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:26.681545 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:26.681981 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:26.682523 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:26.683286 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:26.683492 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:26.683558 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:26.685937 (+ 2379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35181: BlockingRecv error: recv got EOF from 127.0.98.2:35181 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 27 02:34:26 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:26 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:26.861270 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:26.855810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.3:43399)
0927 02:34:26.856163 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:26.856176 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:26.856281 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:26.856532 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:26.856535 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:26.856707 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:26.856806 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:26.856811 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:26.857602 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:26.857605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:26.858211 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:26.858221 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:26.858755 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:26.858758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:26.858828 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:26.859329 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:26.859341 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:26.861120 (+ 1779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":103,"threads_started":1}
W20260927 02:34:26.861361 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:26.855917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43399 (local address 127.0.98.62:37517)
0927 02:34:26.856049 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:26.856052 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:26.856266 (+ 214us) server_negotiation.cc:407] Connection header received
0927 02:34:26.856358 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:26.856360 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:26.856406 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:26.856478 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:26.856925 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:26.857470 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:26.858354 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:26.858580 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:26.858678 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:26.861258 (+ 2580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43399: BlockingRecv error: recv got EOF from 127.0.98.3:43399 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 02:34:27 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:27 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:27.510648 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:27.504607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.1:39151)
0927 02:34:27.504911 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:27.504949 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:27.505054 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:27.505295 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:27.505297 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:27.505515 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:27.505628 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:27.505635 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:27.506615 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:27.506619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:27.507354 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:27.507366 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:27.507934 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:27.507937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:27.508019 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:27.508578 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:27.508593 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:27.510498 (+ 1905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":91,"threads_started":1}
W20260927 02:34:27.510824 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:27.504687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39151 (local address 127.0.98.62:37517)
0927 02:34:27.505048 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:27.505052 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:27.505072 (+ 20us) server_negotiation.cc:407] Connection header received
0927 02:34:27.505136 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:27.505139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:27.505183 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:27.505258 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:27.505819 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:27.506504 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:27.507482 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:27.507815 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:27.507931 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:27.510680 (+ 2749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39151: BlockingRecv error: recv got EOF from 127.0.98.1:39151 (error 108)
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":73,"threads_started":1}
Sep 27 02:34:27 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:27 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:27.691748 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:27.686726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.2:33583)
0927 02:34:27.686996 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:27.687011 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:27.687121 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:27.687351 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:27.687354 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:27.687544 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:27.687631 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:27.687637 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:27.688342 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:27.688345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:27.689115 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:27.689126 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:27.689603 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:27.689606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:27.689668 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:27.690092 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:27.690102 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:27.691641 (+ 1539us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":88,"threads_started":1}
W20260927 02:34:27.691881 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:27.686882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33583 (local address 127.0.98.62:37517)
0927 02:34:27.687013 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:27.687017 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:27.687066 (+ 49us) server_negotiation.cc:407] Connection header received
0927 02:34:27.687204 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:27.687207 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:27.687251 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:27.687314 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:27.687746 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:27.688246 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:27.689223 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:27.689510 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:27.689597 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:27.691773 (+ 2176us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33583: BlockingRecv error: recv got EOF from 127.0.98.2:33583 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 02:34:27 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:27 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:27.866789 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:27.862006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.3:43539)
0927 02:34:27.862235 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:27.862247 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:27.862340 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:27.862583 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:27.862586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:27.862744 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:27.862851 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:27.862856 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:27.863554 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:27.863557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:27.864125 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:27.864132 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:27.864482 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:27.864485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:27.864553 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:27.865027 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:27.865038 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:27.866666 (+ 1628us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":60,"threads_started":1}
W20260927 02:34:27.866904 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:27.862102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43539 (local address 127.0.98.62:37517)
0927 02:34:27.862235 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:27.862239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:27.862303 (+ 64us) server_negotiation.cc:407] Connection header received
0927 02:34:27.862416 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:27.862418 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:27.862460 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:27.862517 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:27.862937 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:27.863442 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:27.864205 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:27.864406 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:27.864469 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:27.866790 (+ 2321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43539: BlockingRecv error: recv got EOF from 127.0.98.3:43539 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 02:34:28 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:28 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:28.516213 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:28.511353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.1:46115)
0927 02:34:28.511612 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:28.511637 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:28.511741 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:28.511953 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:28.511955 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:28.512120 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:28.512213 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:28.512218 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:28.512878 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:28.512880 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:28.513450 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:28.513456 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:28.513885 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:28.513889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:28.513969 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:28.514435 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:28.514446 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:28.516057 (+ 1611us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":81,"threads_started":1}
W20260927 02:34:28.516306 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:28.511395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46115 (local address 127.0.98.62:37517)
0927 02:34:28.511665 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:28.511669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:28.511686 (+ 17us) server_negotiation.cc:407] Connection header received
0927 02:34:28.511819 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:28.511821 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:28.511862 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:28.511930 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:28.512298 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:28.512805 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:28.513531 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:28.513743 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:28.513841 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:28.516182 (+ 2341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46115: BlockingRecv error: recv got EOF from 127.0.98.1:46115 (error 108)
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":52,"threads_started":1}
Sep 27 02:34:28 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:28 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:28.697001 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:28.692397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.2:47495)
0927 02:34:28.692631 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:28.692643 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:28.692744 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:28.692948 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:28.692951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:28.693122 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:28.693229 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:28.693235 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:28.693902 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:28.693903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:28.694438 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:28.694444 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:28.694789 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:28.694791 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:28.694845 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:28.695266 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:28.695276 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:28.696857 (+ 1581us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":72,"threads_started":1}
W20260927 02:34:28.697096 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:28.692443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47495 (local address 127.0.98.62:37517)
0927 02:34:28.692564 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:28.692568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:28.692700 (+ 132us) server_negotiation.cc:407] Connection header received
0927 02:34:28.692812 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:28.692814 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:28.692855 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:28.692917 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:28.693314 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:28.693827 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:28.694513 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:28.694717 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:28.694780 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:28.696998 (+ 2218us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47495: BlockingRecv error: recv got EOF from 127.0.98.2:47495 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 27 02:34:28 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:28 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:28.872382 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:28.867516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.3:54281)
0927 02:34:28.867760 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:28.867773 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:28.867870 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:28.868065 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:28.868066 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:28.868232 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:28.868324 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:28.868329 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:28.869010 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:28.869011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:28.869574 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:28.869581 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:28.869942 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:28.869944 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:28.869999 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:28.870453 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:28.870466 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:28.872261 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":76,"threads_started":1}
W20260927 02:34:28.872475 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:28.867601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54281 (local address 127.0.98.62:37517)
0927 02:34:28.867732 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:28.867743 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:28.867834 (+ 91us) server_negotiation.cc:407] Connection header received
0927 02:34:28.867938 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:28.867940 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:28.867984 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:28.868054 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:28.868403 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:28.868914 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:28.869654 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:28.869867 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:28.869932 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:28.872375 (+ 2443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54281: BlockingRecv error: recv got EOF from 127.0.98.3:54281 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 02:34:29 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:29 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:29.526393 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:29.517071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.1:56895)
0927 02:34:29.517356 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:29.517370 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:29.517479 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:29.517753 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:29.517756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:29.517957 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:29.518060 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:29.518067 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:29.518763 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:29.518765 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:29.519309 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:29.519316 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:29.519701 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:29.519704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:29.519781 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:29.524512 (+ 4731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:29.524524 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:29.526280 (+ 1756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":88,"threads_started":1}
W20260927 02:34:29.526549 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:29.517125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56895 (local address 127.0.98.62:37517)
0927 02:34:29.517362 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:29.517366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:29.517440 (+ 74us) server_negotiation.cc:407] Connection header received
0927 02:34:29.517569 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:29.517572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:29.517626 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:29.517714 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:29.518155 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:29.518685 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:29.519387 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:29.519594 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:29.519671 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:29.526421 (+ 6750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56895: BlockingRecv error: recv got EOF from 127.0.98.1:56895 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":66,"threads_started":1}
Sep 27 02:34:29 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:34:29 dist-test-slave-grlk krb5kdc[8530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:34:29.703270 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:29.697932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.2:35877)
0927 02:34:29.698190 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:29.698204 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:29.698286 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:29.698612 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:29.698615 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:29.698833 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:29.698940 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:29.698949 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:29.699895 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:29.699898 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:29.700501 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:29.700510 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:29.700914 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:29.700917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:29.701023 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:29.701456 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:29.701467 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:29.703116 (+ 1649us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":81,"threads_started":1}
W20260927 02:34:29.703377 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:29.698057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35877 (local address 127.0.98.62:37517)
0927 02:34:29.698237 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:29.698240 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:29.698257 (+ 17us) server_negotiation.cc:407] Connection header received
0927 02:34:29.698365 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:29.698368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:29.698410 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:29.698500 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:29.699064 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:29.699786 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:29.700610 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:29.700830 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:29.700905 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:29.703281 (+ 2376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35877: BlockingRecv error: recv got EOF from 127.0.98.2:35877 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
I20260927 02:34:29.777596 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 9113
I20260927 02:34:29.782198 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 9265
I20260927 02:34:29.786345 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 9397
I20260927 02:34:29.790390 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 9043
2026-09-27T02:34:29Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17767 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T02:34:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:34:29Z Disabled control of system clock
I20260927 02:34:29.817425 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39501
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39501
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 02:34:29.921967 9594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:34:29.922209 9594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:34:29.922240 9594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:34:29.926129 9594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:34:29.926172 9594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:34:29.926186 9594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:34:29.926200 9594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:34:29.926214 9594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:34:29.930711 9594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39501
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:34:29.931618 9594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:34:29.932828 9594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:34:29.940814 9594 server_base.cc:1034] running on GCE node
W20260927 02:34:29.940891 9599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:29.940920 9600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:29.941125 9602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:29.941612 9594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:34:29.942582 9594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:34:29.943753 9594 hybrid_clock.cc:648] HybridClock initialized: now 1790476469943725 us; error 38 us; skew 500 ppm
I20260927 02:34:29.945351 9594 webserver.cc:460] Webserver started at http://127.0.98.62:36815/ using document root <none> and password file <none>
I20260927 02:34:29.945863 9594 fs_manager.cc:359] Metadata directory not provided
I20260927 02:34:29.945924 9594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:34:29.946097 9594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:34:29.948004 9594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "8cbdd92f158c46298e5435f4aa2327cc"
format_stamp: "Formatted at 2026-09-27 02:34:29 on dist-test-slave-grlk"
server_key: "dc0db64171757efa41620447d64b5088"
server_key_iv: "1132335a97811cf0ec3b564b4140fb70"
server_key_version: "encryptionkey@0"
I20260927 02:34:29.948660 9594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "8cbdd92f158c46298e5435f4aa2327cc"
format_stamp: "Formatted at 2026-09-27 02:34:29 on dist-test-slave-grlk"
server_key: "dc0db64171757efa41620447d64b5088"
server_key_iv: "1132335a97811cf0ec3b564b4140fb70"
server_key_version: "encryptionkey@0"
I20260927 02:34:29.952204 9594 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 02:34:29.954490 9608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:29.955619 9594 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:34:29.955725 9594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "8cbdd92f158c46298e5435f4aa2327cc"
format_stamp: "Formatted at 2026-09-27 02:34:29 on dist-test-slave-grlk"
server_key: "dc0db64171757efa41620447d64b5088"
server_key_iv: "1132335a97811cf0ec3b564b4140fb70"
server_key_version: "encryptionkey@0"
I20260927 02:34:29.955812 9594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:34:29.966013 9594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:34:29.966583 9594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:34:29.966737 9594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:34:29.972867 9594 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39501
I20260927 02:34:29.972880 9660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39501 every 8 connection(s)
I20260927 02:34:29.973742 9594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:34:29.976464 9661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:34:29.981505 9661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cbdd92f158c46298e5435f4aa2327cc: Bootstrap starting.
I20260927 02:34:29.982115 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 9594
I20260927 02:34:29.982200 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:34:29.982445 392 external_mini_cluster.cc:1428] Setting key f6279c6b5b5f54d06b482e6dfc617aa2
I20260927 02:34:29.983952 9661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8cbdd92f158c46298e5435f4aa2327cc: Neither blocks nor log segments found. Creating new log.
I20260927 02:34:29.984614 9661 log.cc:824] T 00000000000000000000000000000000 P 8cbdd92f158c46298e5435f4aa2327cc: Log is configured to *not* fsync() on all Append() calls
I20260927 02:34:29.986506 9661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cbdd92f158c46298e5435f4aa2327cc: No bootstrap required, opened a new log
I20260927 02:34:29.990393 9661 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8cbdd92f158c46298e5435f4aa2327cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cbdd92f158c46298e5435f4aa2327cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39501 } }
I20260927 02:34:29.990662 9661 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8cbdd92f158c46298e5435f4aa2327cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:34:29.990707 9661 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8cbdd92f158c46298e5435f4aa2327cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cbdd92f158c46298e5435f4aa2327cc, State: Initialized, Role: FOLLOWER
I20260927 02:34:29.991176 9661 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8cbdd92f158c46298e5435f4aa2327cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cbdd92f158c46298e5435f4aa2327cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39501 } }
I20260927 02:34:29.991286 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:29.983573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60504 (local address 127.0.98.62:39501)
0927 02:34:29.984184 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:29.984198 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:29.984242 (+ 44us) server_negotiation.cc:407] Connection header received
0927 02:34:29.984851 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:29.984866 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:29.985300 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:29.985605 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:29.985918 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:29.986904 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:29.988050 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:29.988492 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:29.988597 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:29.989392 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:29.989396 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:29.989408 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:34:29.989760 (+ 352us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:29.989813 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:34:29.989832 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:29.989901 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:29.990033 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 02:34:29.990277 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":129,"threads_started":1}
I20260927 02:34:29.991299 9661 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8cbdd92f158c46298e5435f4aa2327cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:34:29.991407 9661 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8cbdd92f158c46298e5435f4aa2327cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:34:29.991500 9661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8cbdd92f158c46298e5435f4aa2327cc [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:34:29.992305 9661 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8cbdd92f158c46298e5435f4aa2327cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cbdd92f158c46298e5435f4aa2327cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39501 } }
I20260927 02:34:29.992624 9661 leader_election.cc:302] T 00000000000000000000000000000000 P 8cbdd92f158c46298e5435f4aa2327cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8cbdd92f158c46298e5435f4aa2327cc; no voters:
I20260927 02:34:29.992950 9661 leader_election.cc:288] T 00000000000000000000000000000000 P 8cbdd92f158c46298e5435f4aa2327cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:34:29.993072 9666 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8cbdd92f158c46298e5435f4aa2327cc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:34:29.993299 9666 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8cbdd92f158c46298e5435f4aa2327cc [term 1 LEADER]: Becoming Leader. State: Replica: 8cbdd92f158c46298e5435f4aa2327cc, State: Running, Role: LEADER
I20260927 02:34:29.993641 9666 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8cbdd92f158c46298e5435f4aa2327cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cbdd92f158c46298e5435f4aa2327cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39501 } }
I20260927 02:34:29.994036 9661 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8cbdd92f158c46298e5435f4aa2327cc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:34:29.995090 9667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cbdd92f158c46298e5435f4aa2327cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8cbdd92f158c46298e5435f4aa2327cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cbdd92f158c46298e5435f4aa2327cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39501 } } }
I20260927 02:34:29.995198 9667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cbdd92f158c46298e5435f4aa2327cc [sys.catalog]: This master's current role is: LEADER
I20260927 02:34:29.995348 9668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cbdd92f158c46298e5435f4aa2327cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8cbdd92f158c46298e5435f4aa2327cc. Latest consensus state: current_term: 1 leader_uuid: "8cbdd92f158c46298e5435f4aa2327cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cbdd92f158c46298e5435f4aa2327cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39501 } } }
I20260927 02:34:29.995503 9668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cbdd92f158c46298e5435f4aa2327cc [sys.catalog]: This master's current role is: LEADER
I20260927 02:34:29.995612 9675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:34:29.998188 9675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:34:30.002760 9675 catalog_manager.cc:1347] Generated new cluster ID: 0b8eeda1e2344ab28d68635752b13fd4
I20260927 02:34:30.002817 9675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:34:30.010756 9675 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:34:30.011574 9675 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:34:30.019248 9675 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8cbdd92f158c46298e5435f4aa2327cc: Generated new TSK 0
I20260927 02:34:30.019799 9675 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:34:30.027120 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39501
--builtin_ntp_servers=127.0.98.20:42777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 02:34:30.140524 9685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:34:30.140744 9685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:34:30.140779 9685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:34:30.144743 9685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:34:30.144788 9685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:34:30.144848 9685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:34:30.149341 9685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:34:30.150214 9685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:34:30.151468 9685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:34:30.160178 9690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:30.160176 9691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:30.160282 9685 server_base.cc:1034] running on GCE node
W20260927 02:34:30.160214 9693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:30.160784 9685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:34:30.161389 9685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:34:30.162539 9685 hybrid_clock.cc:648] HybridClock initialized: now 1790476470162522 us; error 31 us; skew 500 ppm
I20260927 02:34:30.164132 9685 webserver.cc:460] Webserver started at http://127.0.98.1:45871/ using document root <none> and password file <none>
I20260927 02:34:30.164629 9685 fs_manager.cc:359] Metadata directory not provided
I20260927 02:34:30.164688 9685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:34:30.164867 9685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:34:30.166719 9685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/ts-0/data/instance:
uuid: "122e093005824aada77c306e553dd128"
format_stamp: "Formatted at 2026-09-27 02:34:30 on dist-test-slave-grlk"
server_key: "5819c04916a03cc8bfe30ac19be81ad3"
server_key_iv: "1ceb9d133ad5e77ddc43b3b34ff737ae"
server_key_version: "encryptionkey@0"
I20260927 02:34:30.167218 9685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/ts-0/wal/instance:
uuid: "122e093005824aada77c306e553dd128"
format_stamp: "Formatted at 2026-09-27 02:34:30 on dist-test-slave-grlk"
server_key: "5819c04916a03cc8bfe30ac19be81ad3"
server_key_iv: "1ceb9d133ad5e77ddc43b3b34ff737ae"
server_key_version: "encryptionkey@0"
I20260927 02:34:30.170619 9685 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 02:34:30.173022 9699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:30.174175 9685 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 02:34:30.174275 9685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/ts-0/wal
uuid: "122e093005824aada77c306e553dd128"
format_stamp: "Formatted at 2026-09-27 02:34:30 on dist-test-slave-grlk"
server_key: "5819c04916a03cc8bfe30ac19be81ad3"
server_key_iv: "1ceb9d133ad5e77ddc43b3b34ff737ae"
server_key_version: "encryptionkey@0"
I20260927 02:34:30.174371 9685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:34:30.191107 9685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:34:30.191725 9685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:34:30.191888 9685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:34:30.192430 9685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:34:30.193337 9685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:34:30.193384 9685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:30.193434 9685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:34:30.193459 9685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:30.201438 9685 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39873
I20260927 02:34:30.201470 9812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39873 every 8 connection(s)
I20260927 02:34:30.202229 9685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:34:30.203182 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 9685
I20260927 02:34:30.203289 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:34:30.203577 392 external_mini_cluster.cc:1428] Setting key 7233ea633c8a16e295c920ebb1c230f9
I20260927 02:34:30.205453 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39501
--builtin_ntp_servers=127.0.98.20:42777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 02:34:30.212217 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:30.203873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45773 (local address 127.0.98.62:39501)
0927 02:34:30.203998 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:30.204003 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:30.205120 (+ 1117us) server_negotiation.cc:407] Connection header received
0927 02:34:30.206546 (+ 1426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:30.206550 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:30.206600 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:30.206683 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:30.207693 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:30.208450 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:30.210090 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:30.210422 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:30.210531 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:30.211748 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:30.211752 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:30.211755 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:34:30.211768 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:30.211791 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:34:30.211802 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:30.211873 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:30.212012 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 02:34:30.212070 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 02:34:30.213002 9815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:30.204272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39501 (local address 127.0.98.1:45773)
0927 02:34:30.204986 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:30.205034 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:30.206321 (+ 1287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:30.206851 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:30.206859 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:30.206960 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:30.207570 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:30.207588 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:30.208576 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:30.208579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:30.209966 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:30.209978 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:30.210581 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:30.210586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:30.210788 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:30.211430 (+ 642us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:34:30.211432 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:30.211488 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:34:30.211570 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:34:30.211882 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:34:30.211889 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:34:30.211895 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 02:34:30.212012 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 02:34:30.212224 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":98,"threads_started":1}
I20260927 02:34:30.214154 9813 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39501
I20260927 02:34:30.214359 9813 heartbeater.cc:464] Registering TS with master...
I20260927 02:34:30.214803 9813 heartbeater.cc:511] Master 127.0.98.62:39501 requested a full tablet report, sending...
I20260927 02:34:30.216125 9625 ts_manager.cc:194] Registered new tserver with Master: 122e093005824aada77c306e553dd128 (127.0.98.1:39873)
I20260927 02:34:30.217356 9625 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:45773
W20260927 02:34:30.310638 9816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:34:30.310866 9816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:34:30.310914 9816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:34:30.314796 9816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:34:30.314841 9816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:34:30.314908 9816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:34:30.319423 9816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:34:30.320365 9816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:34:30.321612 9816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:34:30.330440 9821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:30.330448 9822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:30.330476 9824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:30.330768 9816 server_base.cc:1034] running on GCE node
I20260927 02:34:30.331090 9816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:34:30.331622 9816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:34:30.332801 9816 hybrid_clock.cc:648] HybridClock initialized: now 1790476470332783 us; error 27 us; skew 500 ppm
I20260927 02:34:30.334631 9816 webserver.cc:460] Webserver started at http://127.0.98.2:39191/ using document root <none> and password file <none>
I20260927 02:34:30.335160 9816 fs_manager.cc:359] Metadata directory not provided
I20260927 02:34:30.335224 9816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:34:30.335417 9816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:34:30.337435 9816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/ts-1/data/instance:
uuid: "ac291481421e4ba3bb3bb4a16a341144"
format_stamp: "Formatted at 2026-09-27 02:34:30 on dist-test-slave-grlk"
server_key: "44a16c9e3d2d030088d6175933064c41"
server_key_iv: "3cfa89747ca4a0f7e44185b3aa8bcd9f"
server_key_version: "encryptionkey@0"
I20260927 02:34:30.338023 9816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ac291481421e4ba3bb3bb4a16a341144"
format_stamp: "Formatted at 2026-09-27 02:34:30 on dist-test-slave-grlk"
server_key: "44a16c9e3d2d030088d6175933064c41"
server_key_iv: "3cfa89747ca4a0f7e44185b3aa8bcd9f"
server_key_version: "encryptionkey@0"
I20260927 02:34:30.341528 9816 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:34:30.343828 9830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:30.345170 9816 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:34:30.345265 9816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/ts-1/wal
uuid: "ac291481421e4ba3bb3bb4a16a341144"
format_stamp: "Formatted at 2026-09-27 02:34:30 on dist-test-slave-grlk"
server_key: "44a16c9e3d2d030088d6175933064c41"
server_key_iv: "3cfa89747ca4a0f7e44185b3aa8bcd9f"
server_key_version: "encryptionkey@0"
I20260927 02:34:30.345367 9816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:34:30.358592 9816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:34:30.359184 9816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:34:30.359331 9816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:34:30.359843 9816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:34:30.360715 9816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:34:30.360767 9816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:30.360817 9816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:34:30.360843 9816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:30.369894 9816 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35861
I20260927 02:34:30.369925 9943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35861 every 8 connection(s)
I20260927 02:34:30.370728 9816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:34:30.370968 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 9816
I20260927 02:34:30.371079 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:34:30.371389 392 external_mini_cluster.cc:1428] Setting key 6e8b46b41707292aa2fc3d73192c666b
I20260927 02:34:30.373353 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39501
--builtin_ntp_servers=127.0.98.20:42777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 02:34:30.380633 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:30.372394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57125 (local address 127.0.98.62:39501)
0927 02:34:30.372500 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:30.372504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:30.373808 (+ 1304us) server_negotiation.cc:407] Connection header received
0927 02:34:30.375349 (+ 1541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:30.375352 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:30.375407 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:30.375481 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:30.376499 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:30.377123 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:30.378817 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:30.379047 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:30.379120 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:30.380241 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:30.380244 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:30.380246 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:34:30.380257 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:30.380274 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:34:30.380282 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:30.380331 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:30.380479 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 02:34:30.380526 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 02:34:30.381794 9946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:30.372904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39501 (local address 127.0.98.2:57125)
0927 02:34:30.373655 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:30.373708 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:30.375157 (+ 1449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:30.375656 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:30.375664 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:30.375763 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:30.376367 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:30.376383 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:30.377244 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:30.377248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:30.378698 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:30.378709 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:30.379189 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:30.379194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:30.379387 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:30.379963 (+ 576us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:34:30.379965 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:30.380017 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:34:30.380099 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:34:30.380350 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:34:30.380358 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:34:30.380364 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 02:34:30.380489 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 02:34:30.380785 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":98,"threads_started":1}
I20260927 02:34:30.382786 9944 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39501
I20260927 02:34:30.383020 9944 heartbeater.cc:464] Registering TS with master...
I20260927 02:34:30.383610 9944 heartbeater.cc:511] Master 127.0.98.62:39501 requested a full tablet report, sending...
I20260927 02:34:30.384541 9625 ts_manager.cc:194] Registered new tserver with Master: ac291481421e4ba3bb3bb4a16a341144 (127.0.98.2:35861)
I20260927 02:34:30.385098 9625 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:57125
W20260927 02:34:30.478152 9947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:34:30.478381 9947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:34:30.478428 9947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:34:30.482255 9947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:34:30.482298 9947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:34:30.482368 9947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:34:30.487422 9947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:34:30.488507 9947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:34:30.489821 9947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:34:30.498703 9955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:30.498687 9952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:30.498689 9953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:30.498993 9947 server_base.cc:1034] running on GCE node
I20260927 02:34:30.499342 9947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:34:30.499922 9947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:34:30.501068 9947 hybrid_clock.cc:648] HybridClock initialized: now 1790476470501055 us; error 26 us; skew 500 ppm
I20260927 02:34:30.502664 9947 webserver.cc:460] Webserver started at http://127.0.98.3:45561/ using document root <none> and password file <none>
I20260927 02:34:30.503163 9947 fs_manager.cc:359] Metadata directory not provided
I20260927 02:34:30.503221 9947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:34:30.503412 9947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:34:30.505318 9947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/ts-2/data/instance:
uuid: "933eb5f04af2444799af40e847487828"
format_stamp: "Formatted at 2026-09-27 02:34:30 on dist-test-slave-grlk"
server_key: "3311ca93c7251c3baaadb00e8848d5ee"
server_key_iv: "126d1dff3bf973c595415a3b5899465a"
server_key_version: "encryptionkey@0"
I20260927 02:34:30.505815 9947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/ts-2/wal/instance:
uuid: "933eb5f04af2444799af40e847487828"
format_stamp: "Formatted at 2026-09-27 02:34:30 on dist-test-slave-grlk"
server_key: "3311ca93c7251c3baaadb00e8848d5ee"
server_key_iv: "126d1dff3bf973c595415a3b5899465a"
server_key_version: "encryptionkey@0"
I20260927 02:34:30.509088 9947 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:34:30.511291 9961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:30.512422 9947 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:34:30.512537 9947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/ts-2/wal
uuid: "933eb5f04af2444799af40e847487828"
format_stamp: "Formatted at 2026-09-27 02:34:30 on dist-test-slave-grlk"
server_key: "3311ca93c7251c3baaadb00e8848d5ee"
server_key_iv: "126d1dff3bf973c595415a3b5899465a"
server_key_version: "encryptionkey@0"
I20260927 02:34:30.512631 9947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:34:30.533974 9947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:34:30.534534 9947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:34:30.534686 9947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:34:30.535189 9947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:34:30.536024 9947 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:34:30.536070 9947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:30.536119 9947 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:34:30.536145 9947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:30.543848 9947 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46489
I20260927 02:34:30.543879 10074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46489 every 8 connection(s)
I20260927 02:34:30.544699 9947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:34:30.549211 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 9947
I20260927 02:34:30.549291 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476410933508-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:34:30.549496 392 external_mini_cluster.cc:1428] Setting key 193be0b9ed0f361180879a24a262ffc4
I20260927 02:34:30.553097 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:30.546283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59853 (local address 127.0.98.62:39501)
0927 02:34:30.546408 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:30.546412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:30.547387 (+ 975us) server_negotiation.cc:407] Connection header received
0927 02:34:30.548699 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:30.548702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:30.548746 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:30.548814 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:30.549721 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:30.550368 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:30.551427 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:30.551644 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:30.551726 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:30.552646 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:30.552649 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:30.552652 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:34:30.552663 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:30.552681 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:34:30.552689 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:30.552746 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:30.552888 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 02:34:30.552962 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 02:34:30.553846 10077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:30.546684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39501 (local address 127.0.98.3:59853)
0927 02:34:30.547258 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:30.547294 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:30.548528 (+ 1234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:30.548988 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:30.548996 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:30.549095 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:30.549592 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:30.549608 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:30.550475 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:30.550479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:30.551314 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:30.551322 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:30.551777 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:30.551781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:30.551934 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:30.552422 (+ 488us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:34:30.552423 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:30.552458 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:34:30.552514 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:34:30.552757 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:34:30.552765 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:34:30.552775 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 02:34:30.552875 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 02:34:30.553138 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":98,"threads_started":1}
I20260927 02:34:30.554723 10075 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39501
I20260927 02:34:30.554915 10075 heartbeater.cc:464] Registering TS with master...
I20260927 02:34:30.555348 10075 heartbeater.cc:511] Master 127.0.98.62:39501 requested a full tablet report, sending...
I20260927 02:34:30.556226 9625 ts_manager.cc:194] Registered new tserver with Master: 933eb5f04af2444799af40e847487828 (127.0.98.3:46489)
I20260927 02:34:30.556734 9625 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:59853
I20260927 02:34:30.562542 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 02:34:30.564921 9664 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:30.564177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60518 (local address 127.0.98.62:39501)
0927 02:34:30.564309 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:30.564313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:30.564427 (+ 114us) server_negotiation.cc:407] Connection header received
0927 02:34:30.564522 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:30.564524 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:30.564558 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:30.564609 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:30.564813 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60518: BlockingRecv error: recv got EOF from 127.0.0.1:60518 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 02:34:30.565001 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 9685
I20260927 02:34:30.569482 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 9816
I20260927 02:34:30.573611 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 9947
I20260927 02:34:30.577666 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 9594
2026-09-27T02:34:30Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (785 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T02:34:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:34:30Z Disabled control of system clock
I20260927 02:34:30.603585 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40691
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 02:34:30.705262 10086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:34:30.705492 10086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:34:30.705541 10086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:34:30.709388 10086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:34:30.709437 10086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:34:30.709458 10086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:34:30.709479 10086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:34:30.709507 10086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:34:30.714402 10086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40691
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:34:30.715466 10086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:34:30.716776 10086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:34:30.724397 10091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:30.724431 10092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:30.724516 10086 server_base.cc:1034] running on GCE node
W20260927 02:34:30.724449 10094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:30.725201 10086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:34:30.726174 10086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:34:30.727316 10086 hybrid_clock.cc:648] HybridClock initialized: now 1790476470727292 us; error 38 us; skew 500 ppm
I20260927 02:34:30.728914 10086 webserver.cc:460] Webserver started at http://127.0.98.62:41915/ using document root <none> and password file <none>
I20260927 02:34:30.729434 10086 fs_manager.cc:359] Metadata directory not provided
I20260927 02:34:30.729490 10086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:34:30.729641 10086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:34:30.731437 10086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "12f7278edfaa46219c94d0deb8af2203"
format_stamp: "Formatted at 2026-09-27 02:34:30 on dist-test-slave-grlk"
server_key: "3b5372e5ed2680081b5cc26ff2b09d52"
server_key_iv: "5ade48ff7abe4454a50b5cc4f22d7940"
server_key_version: "encryptionkey@0"
I20260927 02:34:30.731907 10086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "12f7278edfaa46219c94d0deb8af2203"
format_stamp: "Formatted at 2026-09-27 02:34:30 on dist-test-slave-grlk"
server_key: "3b5372e5ed2680081b5cc26ff2b09d52"
server_key_iv: "5ade48ff7abe4454a50b5cc4f22d7940"
server_key_version: "encryptionkey@0"
I20260927 02:34:30.735116 10086 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 02:34:30.737280 10100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:30.738471 10086 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:34:30.738565 10086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "12f7278edfaa46219c94d0deb8af2203"
format_stamp: "Formatted at 2026-09-27 02:34:30 on dist-test-slave-grlk"
server_key: "3b5372e5ed2680081b5cc26ff2b09d52"
server_key_iv: "5ade48ff7abe4454a50b5cc4f22d7940"
server_key_version: "encryptionkey@0"
I20260927 02:34:30.738665 10086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:34:30.742825 10086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:34:30.743348 10086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:34:30.743486 10086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:34:30.749436 10086 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40691
I20260927 02:34:30.749449 10152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40691 every 8 connection(s)
I20260927 02:34:30.750244 10086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:34:30.753057 10153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:34:30.757936 10153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12f7278edfaa46219c94d0deb8af2203: Bootstrap starting.
I20260927 02:34:30.758112 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 10086
I20260927 02:34:30.758211 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:34:30.758440 392 external_mini_cluster.cc:1428] Setting key 117958cfc70caa223176e845d89ab778
I20260927 02:34:30.760253 10153 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12f7278edfaa46219c94d0deb8af2203: Neither blocks nor log segments found. Creating new log.
I20260927 02:34:30.760891 10153 log.cc:824] T 00000000000000000000000000000000 P 12f7278edfaa46219c94d0deb8af2203: Log is configured to *not* fsync() on all Append() calls
I20260927 02:34:30.762600 10153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12f7278edfaa46219c94d0deb8af2203: No bootstrap required, opened a new log
I20260927 02:34:30.763415 10156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:30.759607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34992 (local address 127.0.98.62:40691)
0927 02:34:30.760213 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:30.760236 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:30.760268 (+ 32us) server_negotiation.cc:407] Connection header received
0927 02:34:30.760881 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:30.760897 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:30.761282 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:30.761570 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:30.762248 (+ 678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:30.762252 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:30.762263 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:34:30.762273 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:30.762324 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:34:30.762340 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:30.762429 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:30.762541 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 02:34:30.762778 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":134,"threads_started":1}
I20260927 02:34:30.766263 10153 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12f7278edfaa46219c94d0deb8af2203 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f7278edfaa46219c94d0deb8af2203" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40691 } }
I20260927 02:34:30.766465 10153 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12f7278edfaa46219c94d0deb8af2203 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:34:30.766510 10153 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12f7278edfaa46219c94d0deb8af2203 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12f7278edfaa46219c94d0deb8af2203, State: Initialized, Role: FOLLOWER
I20260927 02:34:30.766888 10153 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12f7278edfaa46219c94d0deb8af2203 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f7278edfaa46219c94d0deb8af2203" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40691 } }
I20260927 02:34:30.767031 10153 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12f7278edfaa46219c94d0deb8af2203 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:34:30.767095 10153 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12f7278edfaa46219c94d0deb8af2203 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:34:30.767185 10153 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12f7278edfaa46219c94d0deb8af2203 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:34:30.768021 10153 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12f7278edfaa46219c94d0deb8af2203 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f7278edfaa46219c94d0deb8af2203" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40691 } }
I20260927 02:34:30.768319 10153 leader_election.cc:302] T 00000000000000000000000000000000 P 12f7278edfaa46219c94d0deb8af2203 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12f7278edfaa46219c94d0deb8af2203; no voters:
I20260927 02:34:30.768577 10153 leader_election.cc:288] T 00000000000000000000000000000000 P 12f7278edfaa46219c94d0deb8af2203 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:34:30.768675 10158 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12f7278edfaa46219c94d0deb8af2203 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:34:30.768882 10158 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12f7278edfaa46219c94d0deb8af2203 [term 1 LEADER]: Becoming Leader. State: Replica: 12f7278edfaa46219c94d0deb8af2203, State: Running, Role: LEADER
I20260927 02:34:30.769224 10158 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12f7278edfaa46219c94d0deb8af2203 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f7278edfaa46219c94d0deb8af2203" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40691 } }
I20260927 02:34:30.769690 10153 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12f7278edfaa46219c94d0deb8af2203 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:34:30.770710 10160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12f7278edfaa46219c94d0deb8af2203 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12f7278edfaa46219c94d0deb8af2203. Latest consensus state: current_term: 1 leader_uuid: "12f7278edfaa46219c94d0deb8af2203" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f7278edfaa46219c94d0deb8af2203" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40691 } } }
I20260927 02:34:30.770869 10160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12f7278edfaa46219c94d0deb8af2203 [sys.catalog]: This master's current role is: LEADER
I20260927 02:34:30.770721 10159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12f7278edfaa46219c94d0deb8af2203 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12f7278edfaa46219c94d0deb8af2203" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f7278edfaa46219c94d0deb8af2203" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40691 } } }
I20260927 02:34:30.771140 10159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12f7278edfaa46219c94d0deb8af2203 [sys.catalog]: This master's current role is: LEADER
I20260927 02:34:30.771232 10167 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:34:30.774082 10167 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:34:30.778668 10167 catalog_manager.cc:1347] Generated new cluster ID: c558043807af43f783391a6715abdaf1
I20260927 02:34:30.778733 10167 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:34:30.788463 10167 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:34:30.788709 10167 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:34:30.793671 10167 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12f7278edfaa46219c94d0deb8af2203: Generated new TSK 0
I20260927 02:34:30.794217 10167 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:34:30.798369 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40691
--builtin_ntp_servers=127.0.98.20:33773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 02:34:30.905797 10177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:34:30.906036 10177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:34:30.906075 10177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:34:30.910015 10177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:34:30.910059 10177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:34:30.910125 10177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:34:30.914822 10177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:34:30.915799 10177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:34:30.917478 10177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:34:30.926357 10182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:30.926402 10185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:30.926435 10177 server_base.cc:1034] running on GCE node
W20260927 02:34:30.926359 10183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:30.926894 10177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:34:30.927459 10177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:34:30.928596 10177 hybrid_clock.cc:648] HybridClock initialized: now 1790476470928578 us; error 32 us; skew 500 ppm
I20260927 02:34:30.930238 10177 webserver.cc:460] Webserver started at http://127.0.98.1:44451/ using document root <none> and password file <none>
I20260927 02:34:30.930734 10177 fs_manager.cc:359] Metadata directory not provided
I20260927 02:34:30.930791 10177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:34:30.930964 10177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:34:30.932797 10177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/ts-0/data/instance:
uuid: "20635f2c4f9044c49673b294b5251f92"
format_stamp: "Formatted at 2026-09-27 02:34:30 on dist-test-slave-grlk"
server_key: "cbc149465e0a85d38e94764b84b7ac54"
server_key_iv: "7c7ea2ec59f16dda80997e9fc5bb4453"
server_key_version: "encryptionkey@0"
I20260927 02:34:30.933319 10177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/ts-0/wal/instance:
uuid: "20635f2c4f9044c49673b294b5251f92"
format_stamp: "Formatted at 2026-09-27 02:34:30 on dist-test-slave-grlk"
server_key: "cbc149465e0a85d38e94764b84b7ac54"
server_key_iv: "7c7ea2ec59f16dda80997e9fc5bb4453"
server_key_version: "encryptionkey@0"
I20260927 02:34:30.936491 10177 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:34:30.938580 10191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:30.939653 10177 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:34:30.939750 10177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/ts-0/wal
uuid: "20635f2c4f9044c49673b294b5251f92"
format_stamp: "Formatted at 2026-09-27 02:34:30 on dist-test-slave-grlk"
server_key: "cbc149465e0a85d38e94764b84b7ac54"
server_key_iv: "7c7ea2ec59f16dda80997e9fc5bb4453"
server_key_version: "encryptionkey@0"
I20260927 02:34:30.939834 10177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:34:30.944031 10177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:34:30.944562 10177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:34:30.944703 10177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:34:30.945215 10177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:34:30.946020 10177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:34:30.946072 10177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:30.946126 10177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:34:30.946151 10177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:30.953868 10177 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38873
I20260927 02:34:30.953896 10304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38873 every 8 connection(s)
I20260927 02:34:30.954666 10177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:34:30.960036 10156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:30.956252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59655 (local address 127.0.98.62:40691)
0927 02:34:30.956372 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:30.956376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:30.957358 (+ 982us) server_negotiation.cc:407] Connection header received
0927 02:34:30.958569 (+ 1211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:30.958572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:30.958616 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:30.958700 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:30.959638 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:30.959642 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:30.959645 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:34:30.959657 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:30.959676 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:34:30.959684 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:30.959758 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:30.959868 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 02:34:30.959908 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 02:34:30.960801 10307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:30.956602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40691 (local address 127.0.98.1:59655)
0927 02:34:30.957233 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:30.957270 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:30.958379 (+ 1109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:30.958849 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:30.958855 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:30.958938 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:30.959401 (+ 463us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:34:30.959402 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:30.959438 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:34:30.959501 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:34:30.959749 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:34:30.959756 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:34:30.959762 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 02:34:30.959903 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 02:34:30.960114 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":110,"threads_started":1}
I20260927 02:34:30.961864 10305 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40691
I20260927 02:34:30.962050 10305 heartbeater.cc:464] Registering TS with master...
I20260927 02:34:30.962409 10305 heartbeater.cc:511] Master 127.0.98.62:40691 requested a full tablet report, sending...
I20260927 02:34:30.963662 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 10177
I20260927 02:34:30.963683 10117 ts_manager.cc:194] Registered new tserver with Master: 20635f2c4f9044c49673b294b5251f92 (127.0.98.1:38873)
I20260927 02:34:30.963747 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:34:30.964018 392 external_mini_cluster.cc:1428] Setting key e1eb636c7420aff9a4be5c61ae9d867e
I20260927 02:34:30.965823 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40691
--builtin_ntp_servers=127.0.98.20:33773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 02:34:31.069656 10308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:34:31.069898 10308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:34:31.069958 10308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:34:31.074051 10308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:34:31.074102 10308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:34:31.074179 10308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:34:31.078794 10308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:34:31.079713 10308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:34:31.081019 10308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:34:31.090054 10314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:31.090210 10308 server_base.cc:1034] running on GCE node
W20260927 02:34:31.090163 10316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:31.090088 10313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:31.090756 10308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:34:31.091449 10308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:34:31.092597 10308 hybrid_clock.cc:648] HybridClock initialized: now 1790476471092577 us; error 32 us; skew 500 ppm
I20260927 02:34:31.094272 10308 webserver.cc:460] Webserver started at http://127.0.98.2:37437/ using document root <none> and password file <none>
I20260927 02:34:31.094784 10308 fs_manager.cc:359] Metadata directory not provided
I20260927 02:34:31.094846 10308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:34:31.095017 10308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:34:31.097158 10308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/ts-1/data/instance:
uuid: "af1b120da9164e42934892fb87030607"
format_stamp: "Formatted at 2026-09-27 02:34:31 on dist-test-slave-grlk"
server_key: "ae549354b6e63cb768a60b6dd8a86c02"
server_key_iv: "2e53ff472c203b98e8d1b4f49b519fca"
server_key_version: "encryptionkey@0"
I20260927 02:34:31.097671 10308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/ts-1/wal/instance:
uuid: "af1b120da9164e42934892fb87030607"
format_stamp: "Formatted at 2026-09-27 02:34:31 on dist-test-slave-grlk"
server_key: "ae549354b6e63cb768a60b6dd8a86c02"
server_key_iv: "2e53ff472c203b98e8d1b4f49b519fca"
server_key_version: "encryptionkey@0"
I20260927 02:34:31.101117 10308 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:34:31.103338 10322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:31.104475 10308 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:34:31.104573 10308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/ts-1/wal
uuid: "af1b120da9164e42934892fb87030607"
format_stamp: "Formatted at 2026-09-27 02:34:31 on dist-test-slave-grlk"
server_key: "ae549354b6e63cb768a60b6dd8a86c02"
server_key_iv: "2e53ff472c203b98e8d1b4f49b519fca"
server_key_version: "encryptionkey@0"
I20260927 02:34:31.104664 10308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:34:31.108882 10308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:34:31.109542 10308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:34:31.109699 10308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:34:31.110280 10308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:34:31.111282 10308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:34:31.111351 10308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:31.111407 10308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:34:31.111439 10308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:31.122008 10308 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42359
I20260927 02:34:31.122051 10435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42359 every 8 connection(s)
I20260927 02:34:31.123014 10308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:34:31.127357 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 10308
I20260927 02:34:31.127480 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:34:31.127772 392 external_mini_cluster.cc:1428] Setting key 847eb97e9ccc169d428c2147f2824628
I20260927 02:34:31.128815 10156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:31.124789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57873 (local address 127.0.98.62:40691)
0927 02:34:31.124917 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:31.124920 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:31.125995 (+ 1075us) server_negotiation.cc:407] Connection header received
0927 02:34:31.127173 (+ 1178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:31.127176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:31.127210 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:31.127284 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:31.128410 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:31.128414 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:31.128417 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:34:31.128429 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:31.128448 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:34:31.128456 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:31.128530 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:31.128646 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 02:34:31.128686 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 02:34:31.129705 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40691
--builtin_ntp_servers=127.0.98.20:33773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 02:34:31.129829 10438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:31.125202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40691 (local address 127.0.98.2:57873)
0927 02:34:31.125859 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:31.125898 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:31.126997 (+ 1099us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:31.127479 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:31.127488 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:31.127596 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:31.128166 (+ 570us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:34:31.128167 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:31.128206 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:34:31.128273 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:34:31.128524 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:34:31.128531 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:34:31.128537 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 02:34:31.128680 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 02:34:31.128912 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":112,"threads_started":1}
I20260927 02:34:31.130802 10436 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40691
I20260927 02:34:31.131033 10436 heartbeater.cc:464] Registering TS with master...
I20260927 02:34:31.131395 10436 heartbeater.cc:511] Master 127.0.98.62:40691 requested a full tablet report, sending...
I20260927 02:34:31.132319 10116 ts_manager.cc:194] Registered new tserver with Master: af1b120da9164e42934892fb87030607 (127.0.98.2:42359)
W20260927 02:34:31.237228 10439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:34:31.237457 10439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:34:31.237494 10439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:34:31.241343 10439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:34:31.241384 10439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:34:31.241437 10439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:34:31.246160 10439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:34:31.247077 10439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:34:31.248335 10439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:34:31.256974 10444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:31.257018 10447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:31.257018 10445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:31.257100 10439 server_base.cc:1034] running on GCE node
I20260927 02:34:31.257556 10439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:34:31.258134 10439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:34:31.259274 10439 hybrid_clock.cc:648] HybridClock initialized: now 1790476471259251 us; error 36 us; skew 500 ppm
I20260927 02:34:31.260886 10439 webserver.cc:460] Webserver started at http://127.0.98.3:37225/ using document root <none> and password file <none>
I20260927 02:34:31.261410 10439 fs_manager.cc:359] Metadata directory not provided
I20260927 02:34:31.261466 10439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:34:31.261638 10439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:34:31.263526 10439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/ts-2/data/instance:
uuid: "6258ed4fe1f94f64afaad1c05cf4b7f0"
format_stamp: "Formatted at 2026-09-27 02:34:31 on dist-test-slave-grlk"
server_key: "a03d11c6434af4bcf9fc41a46ad960ad"
server_key_iv: "5fcc1c893556d4fcbd10d6ac673f11af"
server_key_version: "encryptionkey@0"
I20260927 02:34:31.264083 10439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6258ed4fe1f94f64afaad1c05cf4b7f0"
format_stamp: "Formatted at 2026-09-27 02:34:31 on dist-test-slave-grlk"
server_key: "a03d11c6434af4bcf9fc41a46ad960ad"
server_key_iv: "5fcc1c893556d4fcbd10d6ac673f11af"
server_key_version: "encryptionkey@0"
I20260927 02:34:31.267357 10439 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 02:34:31.269454 10453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:31.270572 10439 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:34:31.270661 10439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/ts-2/wal
uuid: "6258ed4fe1f94f64afaad1c05cf4b7f0"
format_stamp: "Formatted at 2026-09-27 02:34:31 on dist-test-slave-grlk"
server_key: "a03d11c6434af4bcf9fc41a46ad960ad"
server_key_iv: "5fcc1c893556d4fcbd10d6ac673f11af"
server_key_version: "encryptionkey@0"
I20260927 02:34:31.270785 10439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:34:31.274963 10439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:34:31.275466 10439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:34:31.275600 10439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:34:31.276064 10439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:34:31.276844 10439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:34:31.276886 10439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:31.276947 10439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:34:31.276975 10439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:31.284497 10439 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43661
I20260927 02:34:31.284521 10566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43661 every 8 connection(s)
I20260927 02:34:31.285310 10439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:34:31.290632 10156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:31.287051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37739 (local address 127.0.98.62:40691)
0927 02:34:31.287177 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:31.287180 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:31.288095 (+ 915us) server_negotiation.cc:407] Connection header received
0927 02:34:31.289304 (+ 1209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:31.289306 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:31.289340 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:31.289407 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:31.290310 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:31.290313 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:31.290316 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:34:31.290325 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:31.290343 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:34:31.290350 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:31.290415 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:31.290500 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 02:34:31.290532 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 02:34:31.291364 10569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:31.287391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40691 (local address 127.0.98.3:37739)
0927 02:34:31.287980 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:31.288017 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:31.289142 (+ 1125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:31.289536 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:31.289542 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:31.289634 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:31.290093 (+ 459us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:34:31.290095 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:31.290131 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:34:31.290193 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:34:31.290408 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:34:31.290412 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:34:31.290416 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:34:31.290524 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 02:34:31.290707 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":104,"threads_started":1}
I20260927 02:34:31.292330 10567 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40691
I20260927 02:34:31.292512 10567 heartbeater.cc:464] Registering TS with master...
I20260927 02:34:31.292851 10567 heartbeater.cc:511] Master 127.0.98.62:40691 requested a full tablet report, sending...
I20260927 02:34:31.293798 10116 ts_manager.cc:194] Registered new tserver with Master: 6258ed4fe1f94f64afaad1c05cf4b7f0 (127.0.98.3:43661)
I20260927 02:34:31.295532 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 10439
I20260927 02:34:31.295624 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476410933508-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:34:31.295857 392 external_mini_cluster.cc:1428] Setting key 8a173bec6960de96d3d66b8e40f34a87
I20260927 02:34:31.297465 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 02:34:31.299775 10156 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:34:31.299082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35008 (local address 127.0.98.62:40691)
0927 02:34:31.299197 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:31.299201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:31.299341 (+ 140us) server_negotiation.cc:407] Connection header received
0927 02:34:31.299440 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:31.299442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:31.299470 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:31.299522 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:31.299688 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35008: BlockingRecv error: recv got EOF from 127.0.0.1:35008 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 02:34:31.299971 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 10177
I20260927 02:34:31.305167 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 10308
I20260927 02:34:31.309535 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 10439
I20260927 02:34:31.313560 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 10086
2026-09-27T02:34:31Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (735 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:34:31 dist-test-slave-grlk krb5kdc[10577](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:34:31 dist-test-slave-grlk krb5kdc[10577](info): set up 2 sockets
Sep 27 02:34:31 dist-test-slave-grlk krb5kdc[10577](info): commencing operation
krb5kdc: starting...
W20260927 02:34:32.265463 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/krb5kdc/test-user.keytab.
Sep 27 02:34:32 dist-test-slave-grlk krb5kdc[10577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:34:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:34:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:34:32.372102 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34343
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34343
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:34:32.472954 10590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:34:32.473179 10590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:34:32.473228 10590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:34:32.477512 10590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:34:32.477596 10590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:34:32.477626 10590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:34:32.477646 10590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:34:32.477666 10590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:34:32.485312 10590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:34:32.486654 10590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:34:32.487708 10590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:34:32.495872 10598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:32.495916 10590 server_base.cc:1034] running on GCE node
W20260927 02:34:32.495901 10595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:32.495872 10596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:32.496675 10590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:34:32.497767 10590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:34:32.498916 10590 hybrid_clock.cc:648] HybridClock initialized: now 1790476472498885 us; error 48 us; skew 500 ppm
Sep 27 02:34:32 dist-test-slave-grlk krb5kdc[10577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:34:32.501479 10590 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:34:32.502307 10590 webserver.cc:460] Webserver started at http://127.0.98.62:42741/ using document root <none> and password file <none>
I20260927 02:34:32.502823 10590 fs_manager.cc:359] Metadata directory not provided
I20260927 02:34:32.502887 10590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:34:32.503068 10590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:34:32.505319 10590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "3ea8c48e065448709d4b9b63529b8a76"
format_stamp: "Formatted at 2026-09-27 02:34:32 on dist-test-slave-grlk"
server_key: "1b0e16cfc971cc1e24270b8b533b66d6"
server_key_iv: "d31d0a956a1b0678c6d38610880cf29b"
server_key_version: "encryptionkey@0"
I20260927 02:34:32.505847 10590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "3ea8c48e065448709d4b9b63529b8a76"
format_stamp: "Formatted at 2026-09-27 02:34:32 on dist-test-slave-grlk"
server_key: "1b0e16cfc971cc1e24270b8b533b66d6"
server_key_iv: "d31d0a956a1b0678c6d38610880cf29b"
server_key_version: "encryptionkey@0"
I20260927 02:34:32.509624 10590 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 02:34:32.512365 10605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:32.513657 10590 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:34:32.513779 10590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "3ea8c48e065448709d4b9b63529b8a76"
format_stamp: "Formatted at 2026-09-27 02:34:32 on dist-test-slave-grlk"
server_key: "1b0e16cfc971cc1e24270b8b533b66d6"
server_key_iv: "d31d0a956a1b0678c6d38610880cf29b"
server_key_version: "encryptionkey@0"
I20260927 02:34:32.513901 10590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:34:32.531085 10590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:34:32.536888 10590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:34:32.537184 10590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:34:32.543849 10590 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34343
I20260927 02:34:32.543882 10657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34343 every 8 connection(s)
I20260927 02:34:32.544762 10590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:34:32.547502 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 10590
I20260927 02:34:32.547621 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:34:32.547726 10658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:34:32.547927 392 external_mini_cluster.cc:1428] Setting key 31243ce5e35be6340e0d21a179114cfc
I20260927 02:34:32.554410 10658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ea8c48e065448709d4b9b63529b8a76: Bootstrap starting.
I20260927 02:34:32.557003 10658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ea8c48e065448709d4b9b63529b8a76: Neither blocks nor log segments found. Creating new log.
Sep 27 02:34:32 dist-test-slave-grlk krb5kdc[10577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:34:32.557847 10658 log.cc:824] T 00000000000000000000000000000000 P 3ea8c48e065448709d4b9b63529b8a76: Log is configured to *not* fsync() on all Append() calls
I20260927 02:34:32.559971 10658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ea8c48e065448709d4b9b63529b8a76: No bootstrap required, opened a new log
I20260927 02:34:32.563318 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:32.549392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33554 (local address 127.0.98.62:34343)
0927 02:34:32.549819 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:32.549835 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:32.549869 (+ 34us) server_negotiation.cc:407] Connection header received
0927 02:34:32.550541 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:32.550562 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:32.550915 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:32.551203 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:32.552024 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:32.553215 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:32.554339 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:32.554872 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:32.554989 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:32.557683 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:32.557689 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:32.557702 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:32.557729 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:32.559609 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:32.560049 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:32.560055 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:32.560061 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:32.560195 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:32.560493 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:32.560496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:32.560498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:32.561070 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:32.561265 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:32.561804 (+ 539us) server_negotiation.cc:299] Negotiation successful
0927 02:34:32.562182 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":138,"threads_started":1}
I20260927 02:34:32.564862 10658 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ea8c48e065448709d4b9b63529b8a76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ea8c48e065448709d4b9b63529b8a76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34343 } }
I20260927 02:34:32.565274 10658 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ea8c48e065448709d4b9b63529b8a76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:34:32.565337 10658 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ea8c48e065448709d4b9b63529b8a76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ea8c48e065448709d4b9b63529b8a76, State: Initialized, Role: FOLLOWER
I20260927 02:34:32.565822 10658 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ea8c48e065448709d4b9b63529b8a76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ea8c48e065448709d4b9b63529b8a76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34343 } }
I20260927 02:34:32.565974 10658 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ea8c48e065448709d4b9b63529b8a76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:34:32.566037 10658 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ea8c48e065448709d4b9b63529b8a76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:34:32.566139 10658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ea8c48e065448709d4b9b63529b8a76 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:34:32.567171 10658 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ea8c48e065448709d4b9b63529b8a76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ea8c48e065448709d4b9b63529b8a76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34343 } }
I20260927 02:34:32.567502 10658 leader_election.cc:302] T 00000000000000000000000000000000 P 3ea8c48e065448709d4b9b63529b8a76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ea8c48e065448709d4b9b63529b8a76; no voters:
I20260927 02:34:32.567790 10658 leader_election.cc:288] T 00000000000000000000000000000000 P 3ea8c48e065448709d4b9b63529b8a76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:34:32.567893 10663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ea8c48e065448709d4b9b63529b8a76 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:34:32.568099 10663 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ea8c48e065448709d4b9b63529b8a76 [term 1 LEADER]: Becoming Leader. State: Replica: 3ea8c48e065448709d4b9b63529b8a76, State: Running, Role: LEADER
I20260927 02:34:32.568437 10663 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ea8c48e065448709d4b9b63529b8a76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ea8c48e065448709d4b9b63529b8a76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34343 } }
I20260927 02:34:32.569049 10658 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ea8c48e065448709d4b9b63529b8a76 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:34:32.570200 10665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ea8c48e065448709d4b9b63529b8a76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ea8c48e065448709d4b9b63529b8a76. Latest consensus state: current_term: 1 leader_uuid: "3ea8c48e065448709d4b9b63529b8a76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ea8c48e065448709d4b9b63529b8a76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34343 } } }
I20260927 02:34:32.570206 10664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ea8c48e065448709d4b9b63529b8a76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ea8c48e065448709d4b9b63529b8a76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ea8c48e065448709d4b9b63529b8a76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34343 } } }
I20260927 02:34:32.570344 10665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ea8c48e065448709d4b9b63529b8a76 [sys.catalog]: This master's current role is: LEADER
I20260927 02:34:32.570379 10664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ea8c48e065448709d4b9b63529b8a76 [sys.catalog]: This master's current role is: LEADER
I20260927 02:34:32.571163 10673 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:34:32.574079 10673 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:34:32.580220 10673 catalog_manager.cc:1347] Generated new cluster ID: 834259e2a5ce4c4fa3d6d6a988d2d267
I20260927 02:34:32.580308 10673 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:34:32.594506 10673 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:34:32.595402 10673 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:34:32.602169 10673 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ea8c48e065448709d4b9b63529b8a76: Generated new TSK 0
I20260927 02:34:32.602806 10673 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:34:32.634220 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34343
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:34:32.757934 10684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:34:32.758157 10684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:34:32.758198 10684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:34:32.762136 10684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:34:32.762184 10684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:34:32.762245 10684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:34:32.767268 10684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:34:32.768206 10684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:34:32.769183 10684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:34:32.777949 10689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:32.777954 10692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:32.778091 10684 server_base.cc:1034] running on GCE node
W20260927 02:34:32.777954 10690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:32.778560 10684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:34:32.779153 10684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:34:32.780318 10684 hybrid_clock.cc:648] HybridClock initialized: now 1790476472780252 us; error 81 us; skew 500 ppm
Sep 27 02:34:32 dist-test-slave-grlk krb5kdc[10577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:34:32.782811 10684 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:34:32.783598 10684 webserver.cc:460] Webserver started at http://127.0.98.1:34997/ using document root <none> and password file <none>
I20260927 02:34:32.784090 10684 fs_manager.cc:359] Metadata directory not provided
I20260927 02:34:32.784147 10684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:34:32.784327 10684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:34:32.786248 10684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-0/data/instance:
uuid: "8107adc497e64242bab88e6b3179aea6"
format_stamp: "Formatted at 2026-09-27 02:34:32 on dist-test-slave-grlk"
server_key: "1aa110924ffe92ea1aa5257afe5c06da"
server_key_iv: "28a408d159bfbef3d4d3a2d5094763ae"
server_key_version: "encryptionkey@0"
I20260927 02:34:32.786744 10684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8107adc497e64242bab88e6b3179aea6"
format_stamp: "Formatted at 2026-09-27 02:34:32 on dist-test-slave-grlk"
server_key: "1aa110924ffe92ea1aa5257afe5c06da"
server_key_iv: "28a408d159bfbef3d4d3a2d5094763ae"
server_key_version: "encryptionkey@0"
I20260927 02:34:32.790287 10684 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:34:32.792465 10699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:32.793620 10684 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:34:32.793708 10684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-0/wal
uuid: "8107adc497e64242bab88e6b3179aea6"
format_stamp: "Formatted at 2026-09-27 02:34:32 on dist-test-slave-grlk"
server_key: "1aa110924ffe92ea1aa5257afe5c06da"
server_key_iv: "28a408d159bfbef3d4d3a2d5094763ae"
server_key_version: "encryptionkey@0"
I20260927 02:34:32.793823 10684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:34:32.804160 10684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:34:32.806550 10684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:34:32.806696 10684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:34:32.807188 10684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:34:32.807894 10684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:34:32.807937 10684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:32.807976 10684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:34:32.808004 10684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:32.815958 10684 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42713
I20260927 02:34:32.815984 10812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42713 every 8 connection(s)
I20260927 02:34:32.816761 10684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:34:32.819887 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 10684
I20260927 02:34:32.819977 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:34:32.820221 392 external_mini_cluster.cc:1428] Setting key 308b3ab865d4b8c0308f0f50d4762cf0
Sep 27 02:34:32 dist-test-slave-grlk krb5kdc[10577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:34:32.829217 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:32.818393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48227 (local address 127.0.98.62:34343)
0927 02:34:32.818504 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:32.818507 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:32.819307 (+ 800us) server_negotiation.cc:407] Connection header received
0927 02:34:32.820520 (+ 1213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:32.820524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:32.820573 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:32.820643 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:32.821796 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:32.822403 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:32.823462 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:32.823698 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:32.823782 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:32.826153 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:32.826156 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:32.826158 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:32.826180 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:32.827411 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:32.827838 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:32.827842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:32.827843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:32.827910 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:32.828191 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:32.828194 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:32.828196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:32.828368 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:32.828450 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:32.828885 (+ 435us) server_negotiation.cc:299] Negotiation successful
0927 02:34:32.829023 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 02:34:32.829907 10815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:32.818765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34343 (local address 127.0.98.1:48227)
0927 02:34:32.819176 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:32.819223 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:32.820334 (+ 1111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:32.820812 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:32.820819 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:32.821160 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:32.821666 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:32.821679 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:32.822514 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:32.822516 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:32.823349 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:32.823357 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:32.823826 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:32.823829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:32.823977 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:32.824511 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:32.824525 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:32.826011 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:34:32.827509 (+ 1498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:32.827514 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:32.827527 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:32.827752 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:32.827985 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:32.827988 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:32.827990 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:32.828109 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:32.828460 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:34:32.828467 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:34:32.828679 (+ 212us) client_negotiation.cc:769] Sending connection context
0927 02:34:32.828830 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 02:34:32.829145 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":96,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 02:34:32.831094 10813 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34343
I20260927 02:34:32.831298 10813 heartbeater.cc:464] Registering TS with master...
I20260927 02:34:32.831708 10813 heartbeater.cc:511] Master 127.0.98.62:34343 requested a full tablet report, sending...
I20260927 02:34:32.833179 10621 ts_manager.cc:194] Registered new tserver with Master: 8107adc497e64242bab88e6b3179aea6 (127.0.98.1:42713)
I20260927 02:34:32.834697 10621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48227
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:34:32.847410 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34343
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:34:32.955129 10818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:34:32.955384 10818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:34:32.955451 10818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:34:32.959460 10818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:34:32.959519 10818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:34:32.959600 10818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:34:32.964764 10818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:34:32.965726 10818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:34:32.966651 10818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:34:32.975613 10826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:32.975580 10824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:32.975580 10823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:32.975926 10818 server_base.cc:1034] running on GCE node
I20260927 02:34:32.976243 10818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:34:32.976773 10818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:34:32.977921 10818 hybrid_clock.cc:648] HybridClock initialized: now 1790476472977912 us; error 42 us; skew 500 ppm
Sep 27 02:34:32 dist-test-slave-grlk krb5kdc[10577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:34:32.980264 10818 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:34:32.981019 10818 webserver.cc:460] Webserver started at http://127.0.98.2:44249/ using document root <none> and password file <none>
I20260927 02:34:32.981515 10818 fs_manager.cc:359] Metadata directory not provided
I20260927 02:34:32.981576 10818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:34:32.981753 10818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:34:32.983566 10818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-1/data/instance:
uuid: "96b5796a899f4ea1b5415bdf1e1ca47d"
format_stamp: "Formatted at 2026-09-27 02:34:32 on dist-test-slave-grlk"
server_key: "8f0acd43815d35bffb4643f30805ef67"
server_key_iv: "4bb4672c1bd8d76a3bdf3b9d9538a90e"
server_key_version: "encryptionkey@0"
I20260927 02:34:32.984043 10818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-1/wal/instance:
uuid: "96b5796a899f4ea1b5415bdf1e1ca47d"
format_stamp: "Formatted at 2026-09-27 02:34:32 on dist-test-slave-grlk"
server_key: "8f0acd43815d35bffb4643f30805ef67"
server_key_iv: "4bb4672c1bd8d76a3bdf3b9d9538a90e"
server_key_version: "encryptionkey@0"
I20260927 02:34:32.987360 10818 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260927 02:34:32.989455 10833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:32.990506 10818 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260927 02:34:32.990599 10818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-1/wal
uuid: "96b5796a899f4ea1b5415bdf1e1ca47d"
format_stamp: "Formatted at 2026-09-27 02:34:32 on dist-test-slave-grlk"
server_key: "8f0acd43815d35bffb4643f30805ef67"
server_key_iv: "4bb4672c1bd8d76a3bdf3b9d9538a90e"
server_key_version: "encryptionkey@0"
I20260927 02:34:32.990679 10818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:34:32.999861 10818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:34:33.002517 10818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:34:33.002677 10818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:34:33.003197 10818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:34:33.003959 10818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:34:33.004004 10818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:33.004053 10818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:34:33.004082 10818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:33.011904 10818 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37875
I20260927 02:34:33.011941 10946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37875 every 8 connection(s)
I20260927 02:34:33.012715 10818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 02:34:33 dist-test-slave-grlk krb5kdc[10577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:34:33.022778 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 10818
I20260927 02:34:33.022902 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:34:33.023306 392 external_mini_cluster.cc:1428] Setting key a520e769ab771f95d16c69d9222fc54d
I20260927 02:34:33.025843 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:33.014329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37719 (local address 127.0.98.62:34343)
0927 02:34:33.014480 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:33.014485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:33.015243 (+ 758us) server_negotiation.cc:407] Connection header received
0927 02:34:33.016372 (+ 1129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:33.016374 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:33.016418 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:33.016486 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:33.017597 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:33.018134 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:33.019247 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:33.019464 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:33.019536 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:33.022096 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:33.022100 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:33.022102 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:33.022126 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:33.023741 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:33.024246 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:33.024249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:33.024250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:33.024328 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:33.024696 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:33.024699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:33.024701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:33.024992 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:33.025095 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:33.025560 (+ 465us) server_negotiation.cc:299] Negotiation successful
0927 02:34:33.025677 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 02:34:33.026522 10949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:33.014704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34343 (local address 127.0.98.2:37719)
0927 02:34:33.015104 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:33.015143 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:33.016194 (+ 1051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:33.016627 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:33.016633 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:33.017004 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:33.017480 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:33.017493 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:33.018224 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:33.018226 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:33.019151 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:33.019161 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:33.019645 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:33.019649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:33.019788 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:33.020290 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:33.020305 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:33.021911 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:34:33.023854 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:33.023860 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:33.023874 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:33.024164 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:33.024409 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:33.024412 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:33.024414 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:33.024594 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:33.025098 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:34:33.025102 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:34:33.025321 (+ 219us) client_negotiation.cc:769] Sending connection context
0927 02:34:33.025468 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 02:34:33.025720 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":92,"threads_started":1}
I20260927 02:34:33.027613 10947 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34343
I20260927 02:34:33.027833 10947 heartbeater.cc:464] Registering TS with master...
I20260927 02:34:33.028294 10947 heartbeater.cc:511] Master 127.0.98.62:34343 requested a full tablet report, sending...
I20260927 02:34:33.029362 10621 ts_manager.cc:194] Registered new tserver with Master: 96b5796a899f4ea1b5415bdf1e1ca47d (127.0.98.2:37875)
I20260927 02:34:33.029917 10621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37719
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:34:33.047621 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34343
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:34:33.154296 10952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:34:33.154546 10952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:34:33.154600 10952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:34:33.158810 10952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:34:33.158874 10952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:34:33.158955 10952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:34:33.166357 10952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:34:33.167636 10952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:34:33.168886 10952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:34:33.179889 10957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:33.179915 10960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:34:33.179987 10958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:34:33.180249 10952 server_base.cc:1034] running on GCE node
I20260927 02:34:33.180800 10952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:34:33.181627 10952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:34:33.182811 10952 hybrid_clock.cc:648] HybridClock initialized: now 1790476473182792 us; error 36 us; skew 500 ppm
Sep 27 02:34:33 dist-test-slave-grlk krb5kdc[10577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476473, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:34:33.185969 10952 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:34:33.187006 10952 webserver.cc:460] Webserver started at http://127.0.98.3:35721/ using document root <none> and password file <none>
I20260927 02:34:33.187700 10952 fs_manager.cc:359] Metadata directory not provided
I20260927 02:34:33.187781 10952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:34:33.188004 10952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:34:33.190678 10952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-2/data/instance:
uuid: "586e44aa946f43e3b8bcca4f0d9c6334"
format_stamp: "Formatted at 2026-09-27 02:34:33 on dist-test-slave-grlk"
server_key: "617143fa56b071d63eac20be96cef522"
server_key_iv: "af1745ae79a0498ab46c5483165f5c86"
server_key_version: "encryptionkey@0"
I20260927 02:34:33.191339 10952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-2/wal/instance:
uuid: "586e44aa946f43e3b8bcca4f0d9c6334"
format_stamp: "Formatted at 2026-09-27 02:34:33 on dist-test-slave-grlk"
server_key: "617143fa56b071d63eac20be96cef522"
server_key_iv: "af1745ae79a0498ab46c5483165f5c86"
server_key_version: "encryptionkey@0"
I20260927 02:34:33.195959 10952 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 02:34:33.198745 10967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:33.200156 10952 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:34:33.200270 10952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-2/wal
uuid: "586e44aa946f43e3b8bcca4f0d9c6334"
format_stamp: "Formatted at 2026-09-27 02:34:33 on dist-test-slave-grlk"
server_key: "617143fa56b071d63eac20be96cef522"
server_key_iv: "af1745ae79a0498ab46c5483165f5c86"
server_key_version: "encryptionkey@0"
I20260927 02:34:33.200376 10952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:34:33.217164 10952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:34:33.220862 10952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:34:33.221082 10952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:34:33.221771 10952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:34:33.222781 10952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:34:33.222839 10952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:33.222899 10952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:34:33.222929 10952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:34:33.233290 10952 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35823
I20260927 02:34:33.233310 11080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35823 every 8 connection(s)
I20260927 02:34:33.234418 10952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:34:33.243521 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 10952
Sep 27 02:34:33 dist-test-slave-grlk krb5kdc[10577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476473, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:34:33.243635 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:34:33.243916 392 external_mini_cluster.cc:1428] Setting key 4b5b69d07c9a5bfc14860a94bce4df08
I20260927 02:34:33.247483 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:33.236138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46319 (local address 127.0.98.62:34343)
0927 02:34:33.236283 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:33.236289 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:33.237049 (+ 760us) server_negotiation.cc:407] Connection header received
0927 02:34:33.238238 (+ 1189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:33.238242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:33.238296 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:33.238397 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:33.239430 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:33.240271 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:33.241454 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:33.241805 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:33.241912 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:33.244118 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:33.244121 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:33.244123 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:33.244145 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:33.245672 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:33.246129 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:33.246132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:33.246134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:33.246210 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:33.246496 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:33.246499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:33.246500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:33.246669 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:33.246748 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:33.247224 (+ 476us) server_negotiation.cc:299] Negotiation successful
0927 02:34:33.247333 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 02:34:33.248209 11083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:33.236460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34343 (local address 127.0.98.3:46319)
0927 02:34:33.236890 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:33.236960 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:33.238059 (+ 1099us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:33.238521 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:33.238527 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:33.238864 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:34:33.239303 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:33.239316 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:33.240416 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:33.240419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:33.241348 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:33.241358 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:33.241947 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:33.241950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:33.242093 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:33.242633 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:34:33.242648 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:34:33.243966 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:34:33.245767 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:33.245772 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:33.245785 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:33.246048 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:33.246286 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:34:33.246289 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:34:33.246291 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:34:33.246415 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:34:33.246761 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:34:33.246767 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:34:33.246983 (+ 216us) client_negotiation.cc:769] Sending connection context
0927 02:34:33.247169 (+ 186us) client_negotiation.cc:240] Negotiation successful
0927 02:34:33.247457 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":96,"threads_started":1}
I20260927 02:34:33.249194 11081 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34343
I20260927 02:34:33.249396 11081 heartbeater.cc:464] Registering TS with master...
I20260927 02:34:33.249816 11081 heartbeater.cc:511] Master 127.0.98.62:34343 requested a full tablet report, sending...
I20260927 02:34:33.250725 10621 ts_manager.cc:194] Registered new tserver with Master: 586e44aa946f43e3b8bcca4f0d9c6334 (127.0.98.3:35823)
I20260927 02:34:33.251453 10621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46319
I20260927 02:34:33.257726 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:34:33.266844 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:33.259783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33556 (local address 127.0.98.62:34343)
0927 02:34:33.259897 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:33.259901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:33.260037 (+ 136us) server_negotiation.cc:407] Connection header received
0927 02:34:33.260120 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:33.260123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:33.260165 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:33.260242 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:34:33.260802 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:33.261633 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:33.262397 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:33.262719 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:33.262817 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:33.263953 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:34:33.263956 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:34:33.263960 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:34:33.263983 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:34:33.265451 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:33.265797 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:33.265801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:33.265802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:33.265882 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:34:33.266083 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:34:33.266086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:34:33.266088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:34:33.266333 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:34:33.266456 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:33.266591 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 02:34:33.266695 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 02:34:33.269385 10621 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33556:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:34:33.272312 10621 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:34:33.286599 11092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:33.280808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35823 (local address 127.0.0.1:46610)
0927 02:34:33.281291 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:33.281335 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:33.281450 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:33.282386 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:33.282393 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:33.282417 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:34:33.282544 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:33.282551 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:33.284000 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:33.284004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:33.285014 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:33.285025 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:33.286061 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:33.286065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:33.286184 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:33.286225 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 02:34:33.286333 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 02:34:33.286444 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":88,"threads_started":1}
I20260927 02:34:33.286609 11094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:33.281218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46610 (local address 127.0.98.3:35823)
0927 02:34:33.281728 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:33.281740 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:33.281763 (+ 23us) server_negotiation.cc:407] Connection header received
0927 02:34:33.281836 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:33.281841 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:33.282036 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:33.282195 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:34:33.282674 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:33.283874 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:33.285150 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:33.285939 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:33.286042 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:33.286179 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:33.286372 (+ 193us) server_negotiation.cc:299] Negotiation successful
0927 02:34:33.286465 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":141,"threads_started":1}
I20260927 02:34:33.287186 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:33.280839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42713 (local address 127.0.0.1:54658)
0927 02:34:33.282412 (+ 1573us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:33.282426 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:33.282504 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:33.283212 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:33.283215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:33.283231 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:34:33.283323 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:33.283329 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:33.284339 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:33.284343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:33.285589 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:33.285600 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:33.286817 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:33.286821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:33.286914 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:33.286936 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 02:34:33.286994 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 02:34:33.287054 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1513,"thread_start_us":126,"threads_started":1}
I20260927 02:34:33.287684 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:33.281947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54658 (local address 127.0.98.1:42713)
0927 02:34:33.282228 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:33.282239 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:33.282873 (+ 634us) server_negotiation.cc:407] Connection header received
0927 02:34:33.282946 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:33.282952 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:33.283091 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:33.283221 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:34:33.283411 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:33.284207 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:33.285724 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:33.286456 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:33.286551 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:33.287410 (+ 859us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:33.287461 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 02:34:33.287549 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":95,"threads_started":1}
I20260927 02:34:33.289713 11015 tablet_service.cc:1467] Processing CreateTablet for tablet 21fad7ddb2444fe0bd5aa90634f9e88b (DEFAULT_TABLE table=test-table [id=7453eafa7b9e4537864fcfba6864106f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:34:33.290661 11015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21fad7ddb2444fe0bd5aa90634f9e88b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:34:33.292140 11092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:33.281579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37875 (local address 127.0.0.1:56678)
0927 02:34:33.287136 (+ 5557us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:33.287149 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:33.287218 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:33.288015 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:33.288018 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:33.288035 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:34:33.288245 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:33.288251 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:33.289572 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:33.289575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:33.290710 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:33.290721 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:33.291795 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:33.291799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:33.291880 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:33.291903 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 02:34:33.291964 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 02:34:33.292022 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5491,"thread_start_us":86,"threads_started":1}
I20260927 02:34:33.292899 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:33.281772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56678 (local address 127.0.98.2:37875)
0927 02:34:33.282732 (+ 960us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:33.282743 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:33.287458 (+ 4715us) server_negotiation.cc:407] Connection header received
0927 02:34:33.287606 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:33.287611 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:33.287881 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:33.288023 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:34:33.288368 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:33.289442 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:33.290850 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:33.291675 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:33.291781 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:33.292617 (+ 836us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:33.292666 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 02:34:33.292757 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":865,"thread_start_us":92,"threads_started":1}
I20260927 02:34:33.294668 10747 tablet_service.cc:1467] Processing CreateTablet for tablet 21fad7ddb2444fe0bd5aa90634f9e88b (DEFAULT_TABLE table=test-table [id=7453eafa7b9e4537864fcfba6864106f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:34:33.295621 10747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21fad7ddb2444fe0bd5aa90634f9e88b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:34:33.295495 10881 tablet_service.cc:1467] Processing CreateTablet for tablet 21fad7ddb2444fe0bd5aa90634f9e88b (DEFAULT_TABLE table=test-table [id=7453eafa7b9e4537864fcfba6864106f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:34:33.296458 10881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21fad7ddb2444fe0bd5aa90634f9e88b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:34:33.299898 11098 tablet_bootstrap.cc:492] T 21fad7ddb2444fe0bd5aa90634f9e88b P 586e44aa946f43e3b8bcca4f0d9c6334: Bootstrap starting.
I20260927 02:34:33.302104 11098 tablet_bootstrap.cc:654] T 21fad7ddb2444fe0bd5aa90634f9e88b P 586e44aa946f43e3b8bcca4f0d9c6334: Neither blocks nor log segments found. Creating new log.
I20260927 02:34:33.303480 11098 log.cc:824] T 21fad7ddb2444fe0bd5aa90634f9e88b P 586e44aa946f43e3b8bcca4f0d9c6334: Log is configured to *not* fsync() on all Append() calls
I20260927 02:34:33.304610 11101 tablet_bootstrap.cc:492] T 21fad7ddb2444fe0bd5aa90634f9e88b P 96b5796a899f4ea1b5415bdf1e1ca47d: Bootstrap starting.
I20260927 02:34:33.305366 11098 tablet_bootstrap.cc:492] T 21fad7ddb2444fe0bd5aa90634f9e88b P 586e44aa946f43e3b8bcca4f0d9c6334: No bootstrap required, opened a new log
I20260927 02:34:33.305562 11098 ts_tablet_manager.cc:1397] T 21fad7ddb2444fe0bd5aa90634f9e88b P 586e44aa946f43e3b8bcca4f0d9c6334: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 02:34:33.305564 11099 tablet_bootstrap.cc:492] T 21fad7ddb2444fe0bd5aa90634f9e88b P 8107adc497e64242bab88e6b3179aea6: Bootstrap starting.
I20260927 02:34:33.307374 11101 tablet_bootstrap.cc:654] T 21fad7ddb2444fe0bd5aa90634f9e88b P 96b5796a899f4ea1b5415bdf1e1ca47d: Neither blocks nor log segments found. Creating new log.
I20260927 02:34:33.308018 11099 tablet_bootstrap.cc:654] T 21fad7ddb2444fe0bd5aa90634f9e88b P 8107adc497e64242bab88e6b3179aea6: Neither blocks nor log segments found. Creating new log.
I20260927 02:34:33.308267 11101 log.cc:824] T 21fad7ddb2444fe0bd5aa90634f9e88b P 96b5796a899f4ea1b5415bdf1e1ca47d: Log is configured to *not* fsync() on all Append() calls
I20260927 02:34:33.308780 11099 log.cc:824] T 21fad7ddb2444fe0bd5aa90634f9e88b P 8107adc497e64242bab88e6b3179aea6: Log is configured to *not* fsync() on all Append() calls
I20260927 02:34:33.309420 11098 raft_consensus.cc:348] T 21fad7ddb2444fe0bd5aa90634f9e88b P 586e44aa946f43e3b8bcca4f0d9c6334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "586e44aa946f43e3b8bcca4f0d9c6334" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35823 } } peers { permanent_uuid: "8107adc497e64242bab88e6b3179aea6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42713 } } peers { permanent_uuid: "96b5796a899f4ea1b5415bdf1e1ca47d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 } }
I20260927 02:34:33.309689 11098 raft_consensus.cc:374] T 21fad7ddb2444fe0bd5aa90634f9e88b P 586e44aa946f43e3b8bcca4f0d9c6334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:34:33.309743 11098 raft_consensus.cc:729] T 21fad7ddb2444fe0bd5aa90634f9e88b P 586e44aa946f43e3b8bcca4f0d9c6334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 586e44aa946f43e3b8bcca4f0d9c6334, State: Initialized, Role: FOLLOWER
I20260927 02:34:33.310161 11101 tablet_bootstrap.cc:492] T 21fad7ddb2444fe0bd5aa90634f9e88b P 96b5796a899f4ea1b5415bdf1e1ca47d: No bootstrap required, opened a new log
I20260927 02:34:33.310177 11098 consensus_queue.cc:260] T 21fad7ddb2444fe0bd5aa90634f9e88b P 586e44aa946f43e3b8bcca4f0d9c6334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "586e44aa946f43e3b8bcca4f0d9c6334" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35823 } } peers { permanent_uuid: "8107adc497e64242bab88e6b3179aea6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42713 } } peers { permanent_uuid: "96b5796a899f4ea1b5415bdf1e1ca47d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 } }
I20260927 02:34:33.310320 11101 ts_tablet_manager.cc:1397] T 21fad7ddb2444fe0bd5aa90634f9e88b P 96b5796a899f4ea1b5415bdf1e1ca47d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 02:34:33.311086 11098 ts_tablet_manager.cc:1428] T 21fad7ddb2444fe0bd5aa90634f9e88b P 586e44aa946f43e3b8bcca4f0d9c6334: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 02:34:33.311239 11099 tablet_bootstrap.cc:492] T 21fad7ddb2444fe0bd5aa90634f9e88b P 8107adc497e64242bab88e6b3179aea6: No bootstrap required, opened a new log
I20260927 02:34:33.311388 11099 ts_tablet_manager.cc:1397] T 21fad7ddb2444fe0bd5aa90634f9e88b P 8107adc497e64242bab88e6b3179aea6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 02:34:33.311393 11081 heartbeater.cc:503] Master 127.0.98.62:34343 was elected leader, sending a full tablet report...
I20260927 02:34:33.313756 11101 raft_consensus.cc:348] T 21fad7ddb2444fe0bd5aa90634f9e88b P 96b5796a899f4ea1b5415bdf1e1ca47d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "586e44aa946f43e3b8bcca4f0d9c6334" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35823 } } peers { permanent_uuid: "8107adc497e64242bab88e6b3179aea6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42713 } } peers { permanent_uuid: "96b5796a899f4ea1b5415bdf1e1ca47d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 } }
I20260927 02:34:33.314010 11101 raft_consensus.cc:374] T 21fad7ddb2444fe0bd5aa90634f9e88b P 96b5796a899f4ea1b5415bdf1e1ca47d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:34:33.314061 11101 raft_consensus.cc:729] T 21fad7ddb2444fe0bd5aa90634f9e88b P 96b5796a899f4ea1b5415bdf1e1ca47d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96b5796a899f4ea1b5415bdf1e1ca47d, State: Initialized, Role: FOLLOWER
I20260927 02:34:33.314698 11101 consensus_queue.cc:260] T 21fad7ddb2444fe0bd5aa90634f9e88b P 96b5796a899f4ea1b5415bdf1e1ca47d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "586e44aa946f43e3b8bcca4f0d9c6334" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35823 } } peers { permanent_uuid: "8107adc497e64242bab88e6b3179aea6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42713 } } peers { permanent_uuid: "96b5796a899f4ea1b5415bdf1e1ca47d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 } }
I20260927 02:34:33.314996 11099 raft_consensus.cc:348] T 21fad7ddb2444fe0bd5aa90634f9e88b P 8107adc497e64242bab88e6b3179aea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "586e44aa946f43e3b8bcca4f0d9c6334" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35823 } } peers { permanent_uuid: "8107adc497e64242bab88e6b3179aea6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42713 } } peers { permanent_uuid: "96b5796a899f4ea1b5415bdf1e1ca47d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 } }
I20260927 02:34:33.315251 11099 raft_consensus.cc:374] T 21fad7ddb2444fe0bd5aa90634f9e88b P 8107adc497e64242bab88e6b3179aea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:34:33.315302 11099 raft_consensus.cc:729] T 21fad7ddb2444fe0bd5aa90634f9e88b P 8107adc497e64242bab88e6b3179aea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8107adc497e64242bab88e6b3179aea6, State: Initialized, Role: FOLLOWER
I20260927 02:34:33.315716 11099 consensus_queue.cc:260] T 21fad7ddb2444fe0bd5aa90634f9e88b P 8107adc497e64242bab88e6b3179aea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "586e44aa946f43e3b8bcca4f0d9c6334" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35823 } } peers { permanent_uuid: "8107adc497e64242bab88e6b3179aea6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42713 } } peers { permanent_uuid: "96b5796a899f4ea1b5415bdf1e1ca47d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 } }
I20260927 02:34:33.315788 11101 ts_tablet_manager.cc:1428] T 21fad7ddb2444fe0bd5aa90634f9e88b P 96b5796a899f4ea1b5415bdf1e1ca47d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 02:34:33.316162 10947 heartbeater.cc:503] Master 127.0.98.62:34343 was elected leader, sending a full tablet report...
I20260927 02:34:33.316581 11099 ts_tablet_manager.cc:1428] T 21fad7ddb2444fe0bd5aa90634f9e88b P 8107adc497e64242bab88e6b3179aea6: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 02:34:33.316915 10813 heartbeater.cc:503] Master 127.0.98.62:34343 was elected leader, sending a full tablet report...
W20260927 02:34:33.317744 10814 tablet.cc:2369] T 21fad7ddb2444fe0bd5aa90634f9e88b P 8107adc497e64242bab88e6b3179aea6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:34:33.485406 11082 tablet.cc:2369] T 21fad7ddb2444fe0bd5aa90634f9e88b P 586e44aa946f43e3b8bcca4f0d9c6334: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:34:33.487921 11105 raft_consensus.cc:482] T 21fad7ddb2444fe0bd5aa90634f9e88b P 96b5796a899f4ea1b5415bdf1e1ca47d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:34:33.488080 11105 raft_consensus.cc:504] T 21fad7ddb2444fe0bd5aa90634f9e88b P 96b5796a899f4ea1b5415bdf1e1ca47d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "586e44aa946f43e3b8bcca4f0d9c6334" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35823 } } peers { permanent_uuid: "8107adc497e64242bab88e6b3179aea6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42713 } } peers { permanent_uuid: "96b5796a899f4ea1b5415bdf1e1ca47d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 } }
I20260927 02:34:33.489147 11105 leader_election.cc:288] T 21fad7ddb2444fe0bd5aa90634f9e88b P 96b5796a899f4ea1b5415bdf1e1ca47d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 586e44aa946f43e3b8bcca4f0d9c6334 (127.0.98.3:35823), 8107adc497e64242bab88e6b3179aea6 (127.0.98.1:42713)
I20260927 02:34:33.492580 11094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:33.489452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46071 (local address 127.0.98.3:35823)
0927 02:34:33.489564 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:33.489569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:33.489646 (+ 77us) server_negotiation.cc:407] Connection header received
0927 02:34:33.489786 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:33.489789 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:33.489829 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:33.489888 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:34:33.490124 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:33.490602 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:33.491643 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:33.492029 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:33.492124 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:33.492181 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:33.492397 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 02:34:33.492460 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 02:34:33.492604 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:33.489485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37701 (local address 127.0.98.1:42713)
0927 02:34:33.489589 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:33.489593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:33.489776 (+ 183us) server_negotiation.cc:407] Connection header received
0927 02:34:33.489888 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:33.489891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:33.489931 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:33.489990 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:34:33.490224 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:33.490723 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:33.491692 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:33.492064 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:33.492149 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:33.492206 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:33.492418 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 02:34:33.492485 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 02:34:33.492617 10949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:33.489387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35823 (local address 127.0.98.2:46071)
0927 02:34:33.489577 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:33.489590 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:33.489687 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:33.489903 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:33.489905 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:33.489920 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:34:33.490029 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:33.490034 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:33.490696 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:33.490699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:33.491521 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:33.491529 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:33.492135 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:33.492138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:33.492237 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:33.492310 (+ 73us) client_negotiation.cc:769] Sending connection context
0927 02:34:33.492373 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 02:34:33.492458 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":83}
I20260927 02:34:33.492631 11107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:33.489407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42713 (local address 127.0.98.2:37701)
0927 02:34:33.489680 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:33.489691 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:34:33.489793 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:34:33.490004 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:34:33.490007 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:34:33.490021 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:34:33.490126 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:33.490131 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:33.490817 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:33.490820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:33.491580 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:34:33.491587 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:34:33.492157 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:34:33.492160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:34:33.492281 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:33.492329 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 02:34:33.492390 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 02:34:33.492478 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":47,"thread_start_us":76,"threads_started":1}
I20260927 02:34:33.493340 11035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21fad7ddb2444fe0bd5aa90634f9e88b" candidate_uuid: "96b5796a899f4ea1b5415bdf1e1ca47d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "586e44aa946f43e3b8bcca4f0d9c6334" is_pre_election: true
I20260927 02:34:33.493340 10767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21fad7ddb2444fe0bd5aa90634f9e88b" candidate_uuid: "96b5796a899f4ea1b5415bdf1e1ca47d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8107adc497e64242bab88e6b3179aea6" is_pre_election: true
I20260927 02:34:33.493603 11035 raft_consensus.cc:2454] T 21fad7ddb2444fe0bd5aa90634f9e88b P 586e44aa946f43e3b8bcca4f0d9c6334 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96b5796a899f4ea1b5415bdf1e1ca47d in term 0.
I20260927 02:34:33.493603 10767 raft_consensus.cc:2454] T 21fad7ddb2444fe0bd5aa90634f9e88b P 8107adc497e64242bab88e6b3179aea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96b5796a899f4ea1b5415bdf1e1ca47d in term 0.
I20260927 02:34:33.494029 10834 leader_election.cc:302] T 21fad7ddb2444fe0bd5aa90634f9e88b P 96b5796a899f4ea1b5415bdf1e1ca47d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 586e44aa946f43e3b8bcca4f0d9c6334, 96b5796a899f4ea1b5415bdf1e1ca47d; no voters:
I20260927 02:34:33.494277 11105 raft_consensus.cc:2781] T 21fad7ddb2444fe0bd5aa90634f9e88b P 96b5796a899f4ea1b5415bdf1e1ca47d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:34:33.494350 11105 raft_consensus.cc:482] T 21fad7ddb2444fe0bd5aa90634f9e88b P 96b5796a899f4ea1b5415bdf1e1ca47d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:34:33.494400 11105 raft_consensus.cc:3037] T 21fad7ddb2444fe0bd5aa90634f9e88b P 96b5796a899f4ea1b5415bdf1e1ca47d [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:34:33.495389 11105 raft_consensus.cc:504] T 21fad7ddb2444fe0bd5aa90634f9e88b P 96b5796a899f4ea1b5415bdf1e1ca47d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "586e44aa946f43e3b8bcca4f0d9c6334" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35823 } } peers { permanent_uuid: "8107adc497e64242bab88e6b3179aea6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42713 } } peers { permanent_uuid: "96b5796a899f4ea1b5415bdf1e1ca47d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 } }
I20260927 02:34:33.495687 11105 leader_election.cc:288] T 21fad7ddb2444fe0bd5aa90634f9e88b P 96b5796a899f4ea1b5415bdf1e1ca47d [CANDIDATE]: Term 1 election: Requested vote from peers 586e44aa946f43e3b8bcca4f0d9c6334 (127.0.98.3:35823), 8107adc497e64242bab88e6b3179aea6 (127.0.98.1:42713)
I20260927 02:34:33.495975 10767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21fad7ddb2444fe0bd5aa90634f9e88b" candidate_uuid: "96b5796a899f4ea1b5415bdf1e1ca47d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8107adc497e64242bab88e6b3179aea6"
I20260927 02:34:33.495977 11035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21fad7ddb2444fe0bd5aa90634f9e88b" candidate_uuid: "96b5796a899f4ea1b5415bdf1e1ca47d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "586e44aa946f43e3b8bcca4f0d9c6334"
I20260927 02:34:33.496093 11035 raft_consensus.cc:3037] T 21fad7ddb2444fe0bd5aa90634f9e88b P 586e44aa946f43e3b8bcca4f0d9c6334 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:34:33.496078 10767 raft_consensus.cc:3037] T 21fad7ddb2444fe0bd5aa90634f9e88b P 8107adc497e64242bab88e6b3179aea6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:34:33.497113 11035 raft_consensus.cc:2454] T 21fad7ddb2444fe0bd5aa90634f9e88b P 586e44aa946f43e3b8bcca4f0d9c6334 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96b5796a899f4ea1b5415bdf1e1ca47d in term 1.
I20260927 02:34:33.497151 10767 raft_consensus.cc:2454] T 21fad7ddb2444fe0bd5aa90634f9e88b P 8107adc497e64242bab88e6b3179aea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96b5796a899f4ea1b5415bdf1e1ca47d in term 1.
I20260927 02:34:33.497349 10834 leader_election.cc:302] T 21fad7ddb2444fe0bd5aa90634f9e88b P 96b5796a899f4ea1b5415bdf1e1ca47d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 586e44aa946f43e3b8bcca4f0d9c6334, 96b5796a899f4ea1b5415bdf1e1ca47d; no voters:
I20260927 02:34:33.497488 11105 raft_consensus.cc:2781] T 21fad7ddb2444fe0bd5aa90634f9e88b P 96b5796a899f4ea1b5415bdf1e1ca47d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:34:33.497699 11105 raft_consensus.cc:686] T 21fad7ddb2444fe0bd5aa90634f9e88b P 96b5796a899f4ea1b5415bdf1e1ca47d [term 1 LEADER]: Becoming Leader. State: Replica: 96b5796a899f4ea1b5415bdf1e1ca47d, State: Running, Role: LEADER
I20260927 02:34:33.497974 11105 consensus_queue.cc:237] T 21fad7ddb2444fe0bd5aa90634f9e88b P 96b5796a899f4ea1b5415bdf1e1ca47d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "586e44aa946f43e3b8bcca4f0d9c6334" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35823 } } peers { permanent_uuid: "8107adc497e64242bab88e6b3179aea6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42713 } } peers { permanent_uuid: "96b5796a899f4ea1b5415bdf1e1ca47d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 } }
I20260927 02:34:33.500717 10618 catalog_manager.cc:5495] T 21fad7ddb2444fe0bd5aa90634f9e88b P 96b5796a899f4ea1b5415bdf1e1ca47d reported cstate change: term changed from 0 to 1, leader changed from <none> to 96b5796a899f4ea1b5415bdf1e1ca47d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "96b5796a899f4ea1b5415bdf1e1ca47d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "586e44aa946f43e3b8bcca4f0d9c6334" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8107adc497e64242bab88e6b3179aea6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96b5796a899f4ea1b5415bdf1e1ca47d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37875 } health_report { overall_health: HEALTHY } } }
W20260927 02:34:33.513731 10948 tablet.cc:2369] T 21fad7ddb2444fe0bd5aa90634f9e88b P 96b5796a899f4ea1b5415bdf1e1ca47d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:34:33.517841 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:33.515017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56686 (local address 127.0.98.2:37875)
0927 02:34:33.515140 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 02:34:33.515143 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:34:33.515156 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:34:33.515216 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:34:33.515218 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:34:33.515265 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:34:33.515344 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:34:33.515517 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:33.515987 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:33.516723 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:34:33.516915 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:34:33.517020 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:34:33.517219 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:34:33.517574 (+ 355us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:34:33.517653 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 02:34:33.517680 (+ 27us) server_negotiation.cc:299] Negotiation successful
0927 02:34:33.517724 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 02:34:33.522861 10767 raft_consensus.cc:1261] T 21fad7ddb2444fe0bd5aa90634f9e88b P 8107adc497e64242bab88e6b3179aea6 [term 1 FOLLOWER]: Refusing update from remote peer 96b5796a899f4ea1b5415bdf1e1ca47d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:34:33.522862 11035 raft_consensus.cc:1261] T 21fad7ddb2444fe0bd5aa90634f9e88b P 586e44aa946f43e3b8bcca4f0d9c6334 [term 1 FOLLOWER]: Refusing update from remote peer 96b5796a899f4ea1b5415bdf1e1ca47d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:34:33.523504 11105 consensus_queue.cc:1035] T 21fad7ddb2444fe0bd5aa90634f9e88b P 96b5796a899f4ea1b5415bdf1e1ca47d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8107adc497e64242bab88e6b3179aea6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:34:33.523689 11108 consensus_queue.cc:1035] T 21fad7ddb2444fe0bd5aa90634f9e88b P 96b5796a899f4ea1b5415bdf1e1ca47d [LEADER]: Connected to new peer: Peer: permanent_uuid: "586e44aa946f43e3b8bcca4f0d9c6334" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:34:33.531069 11113 mvcc.cc:205] Tried to move back new op lower bound from 7333791635543375872 to 7333791635449950208. Current Snapshot: MvccSnapshot[applied={T|T < 7333791635543375872}]
I20260927 02:34:33.532999 11112 mvcc.cc:205] Tried to move back new op lower bound from 7333791635543375872 to 7333791635449950208. Current Snapshot: MvccSnapshot[applied={T|T < 7333791635543375872}]
I20260927 02:34:33.539405 10618 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:34:33.539574 10618 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:34:33.541781 10618 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3ea8c48e065448709d4b9b63529b8a76: Sending DeleteTablet for 3 replicas of tablet 21fad7ddb2444fe0bd5aa90634f9e88b
I20260927 02:34:33.542517 10881 tablet_service.cc:1514] Processing DeleteTablet for tablet 21fad7ddb2444fe0bd5aa90634f9e88b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:34:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56678
I20260927 02:34:33.542484 11015 tablet_service.cc:1514] Processing DeleteTablet for tablet 21fad7ddb2444fe0bd5aa90634f9e88b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:34:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46610
I20260927 02:34:33.542654 10747 tablet_service.cc:1514] Processing DeleteTablet for tablet 21fad7ddb2444fe0bd5aa90634f9e88b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:34:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54658
I20260927 02:34:33.542924 11121 tablet_replica.cc:330] T 21fad7ddb2444fe0bd5aa90634f9e88b P 96b5796a899f4ea1b5415bdf1e1ca47d: stopping tablet replica
I20260927 02:34:33.543224 11121 raft_consensus.cc:2229] T 21fad7ddb2444fe0bd5aa90634f9e88b P 96b5796a899f4ea1b5415bdf1e1ca47d [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:34:33.543594 11121 raft_consensus.cc:2258] T 21fad7ddb2444fe0bd5aa90634f9e88b P 96b5796a899f4ea1b5415bdf1e1ca47d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:34:33.543675 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 10684
I20260927 02:34:33.545708 11122 tablet_replica.cc:330] T 21fad7ddb2444fe0bd5aa90634f9e88b P 586e44aa946f43e3b8bcca4f0d9c6334: stopping tablet replica
I20260927 02:34:33.545955 11122 raft_consensus.cc:2229] T 21fad7ddb2444fe0bd5aa90634f9e88b P 586e44aa946f43e3b8bcca4f0d9c6334 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:34:33.546204 11122 raft_consensus.cc:2258] T 21fad7ddb2444fe0bd5aa90634f9e88b P 586e44aa946f43e3b8bcca4f0d9c6334 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:34:33.547276 11121 ts_tablet_manager.cc:1905] T 21fad7ddb2444fe0bd5aa90634f9e88b P 96b5796a899f4ea1b5415bdf1e1ca47d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:34:33.549324 11121 ts_tablet_manager.cc:1918] T 21fad7ddb2444fe0bd5aa90634f9e88b P 96b5796a899f4ea1b5415bdf1e1ca47d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:34:33.549401 11121 log.cc:1196] T 21fad7ddb2444fe0bd5aa90634f9e88b P 96b5796a899f4ea1b5415bdf1e1ca47d: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-1/wal/wals/21fad7ddb2444fe0bd5aa90634f9e88b
I20260927 02:34:33.549638 11121 ts_tablet_manager.cc:1939] T 21fad7ddb2444fe0bd5aa90634f9e88b P 96b5796a899f4ea1b5415bdf1e1ca47d: Deleting consensus metadata
I20260927 02:34:33.550501 10607 catalog_manager.cc:4841] TS 96b5796a899f4ea1b5415bdf1e1ca47d (127.0.98.2:37875): tablet 21fad7ddb2444fe0bd5aa90634f9e88b (table test-table [id=7453eafa7b9e4537864fcfba6864106f]) successfully deleted
W20260927 02:34:33.553946 10607 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42713 (error 108)
I20260927 02:34:33.554775 11092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:34:33.554544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42713 (local address 127.0.0.1:54660)
0927 02:34:33.554649 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 02:34:33.554706 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 02:34:33.554877 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 10818
W20260927 02:34:33.555022 10607 catalog_manager.cc:4584] TS 8107adc497e64242bab88e6b3179aea6 (127.0.98.1:42713): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 21fad7ddb2444fe0bd5aa90634f9e88b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42713: connect: Connection refused (error 111)
I20260927 02:34:33.555608 11122 ts_tablet_manager.cc:1905] T 21fad7ddb2444fe0bd5aa90634f9e88b P 586e44aa946f43e3b8bcca4f0d9c6334: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:34:33.557668 11122 ts_tablet_manager.cc:1918] T 21fad7ddb2444fe0bd5aa90634f9e88b P 586e44aa946f43e3b8bcca4f0d9c6334: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:34:33.557765 11122 log.cc:1196] T 21fad7ddb2444fe0bd5aa90634f9e88b P 586e44aa946f43e3b8bcca4f0d9c6334: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476410933508-392-0/minicluster-data/ts-2/wal/wals/21fad7ddb2444fe0bd5aa90634f9e88b
I20260927 02:34:33.558043 11122 ts_tablet_manager.cc:1939] T 21fad7ddb2444fe0bd5aa90634f9e88b P 586e44aa946f43e3b8bcca4f0d9c6334: Deleting consensus metadata
I20260927 02:34:33.559054 10606 catalog_manager.cc:4841] TS 586e44aa946f43e3b8bcca4f0d9c6334 (127.0.98.3:35823): tablet 21fad7ddb2444fe0bd5aa90634f9e88b (table test-table [id=7453eafa7b9e4537864fcfba6864106f]) successfully deleted
I20260927 02:34:33.561017 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 10952
I20260927 02:34:33.566100 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 10590
2026-09-27T02:34:33Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2253 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:34:33 dist-test-slave-grlk krb5kdc[11127](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:34:33 dist-test-slave-grlk krb5kdc[11127](info): set up 2 sockets
Sep 27 02:34:33 dist-test-slave-grlk krb5kdc[11127](info): commencing operation
krb5kdc: starting...
W20260927 02:34:34.509509 392 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/krb5kdc/test-user.keytab.
Sep 27 02:34:34 dist-test-slave-grlk krb5kdc[11127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:34:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:34:34Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 02:34:34.655823 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskcH4HTG/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/postgres -l logfile start
2026-09-27 02:34:36.908 UTC [11157] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 02:34:36.908 UTC [11157] LOG: listening on IPv4 address "127.0.98.20", port 35515
2026-09-27 02:34:36.911 UTC [11157] LOG: listening on Unix socket "/tmp/.s.PGSQL.35515"
2026-09-27 02:34:36.937 UTC [11160] LOG: database system was shut down at 2026-09-27 02:34:35 UTC
2026-09-27 02:34:36.940 UTC [11157] LOG: database system is ready to accept connections
I20260927 02:34:37.809764 392 mini_postgres.cc:96] Postgres bound to 35515
2026-09-27 02:34:37.815 UTC [11168] FATAL: database "slave" does not exist
127.0.98.20:35515 - accepting connections
I20260927 02:34:37.816136 392 mini_ranger.cc:161] Starting Ranger...
I20260927 02:34:37.830626 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 02:34:38.240769 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 02:34:38.240841 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ranger-admin
2026-09-27 02:34:38,497 [I] DB FLAVOR :POSTGRES
2026-09-27 02:34:38,498 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 02:34:38,498 [I] Checking connection..
2026-09-27 02:34:38,498 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcH4HTG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 02:34:38,741 [I] Checking connection passed.
2026-09-27 02:34:38,741 [I] --------- Verifying version history table ---------
2026-09-27 02:34:38,741 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcH4HTG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 02:34:38,968 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 02:34:38,968 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcH4HTG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 02:34:39,207 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 02:34:39,207 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 02:34:39,207 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcH4HTG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35515/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 02:34:39.448 UTC [11232] WARNING: there is no transaction in progress
2026-09-27 02:34:39,453 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 02:34:39,454 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcH4HTG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 02:34:39,680 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 02:34:39,680 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 02:34:39,680 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcH4HTG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 02:34:39,907 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcH4HTG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 02:34:40,141 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcH4HTG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-grlk.c.gcp-upstream.internal','N') ;"
2026-09-27 02:34:40,362 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcH4HTG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 02:34:40,598 [I] Table x_portal_user does not exist in database ranger
2026-09-27 02:34:40,599 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcH4HTG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 02:34:40,847 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 02:34:40,848 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcH4HTG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 02:34:41,071 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcH4HTG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 02:34:41,319 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 02:34:41,319 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcH4HTG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35515/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 02:34:42.196 UTC [11340] WARNING: there is no transaction in progress
2026-09-27 02:34:42.212 UTC [11340] WARNING: there is no transaction in progress
2026-09-27 02:34:42.225 UTC [11340] WARNING: there is no transaction in progress
2026-09-27 02:34:42.238 UTC [11340] WARNING: there is no transaction in progress
2026-09-27 02:34:42.252 UTC [11340] WARNING: there is no transaction in progress
2026-09-27 02:34:42.269 UTC [11340] WARNING: there is no transaction in progress
2026-09-27 02:34:42.320 UTC [11340] WARNING: there is no transaction in progress
2026-09-27 02:34:42.326 UTC [11340] WARNING: there is no transaction in progress
2026-09-27 02:34:42.338 UTC [11340] WARNING: there is no transaction in progress
2026-09-27 02:34:42.347 UTC [11340] WARNING: there is no transaction in progress
2026-09-27 02:34:42.356 UTC [11340] WARNING: there is no transaction in progress
2026-09-27 02:34:42.368 UTC [11340] WARNING: there is no transaction in progress
2026-09-27 02:34:42.377 UTC [11340] WARNING: there is no transaction in progress
2026-09-27 02:34:42.382 UTC [11340] WARNING: there is no transaction in progress
2026-09-27 02:34:42.387 UTC [11340] WARNING: there is no transaction in progress
2026-09-27 02:34:42.392 UTC [11340] WARNING: there is no transaction in progress
2026-09-27 02:34:42,951 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 02:34:42,951 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcH4HTG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 02:34:43,196 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 02:34:43,197 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcH4HTG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 02:34:43,443 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 02:34:43,443 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcH4HTG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 02:34:43,667 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 02:34:43,667 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcH4HTG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 02:34:43,901 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcH4HTG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 02:34:44,148 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcH4HTG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-grlk.c.gcp-upstream.internal';"
2026-09-27 02:34:44,376 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 02:34:44,377 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcH4HTG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35515/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 02:34:44,602 [I] DB_PATCHES have already been applied
I20260927 02:34:44.607676 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskcH4HTG/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskcH4HTG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 02:34:44.607731 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 02:34:44.610689 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:60229
Sep 27, 2026 2:34:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 2:34:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 2:34:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 2:34:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 2:34:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 2:34:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 2:34:45 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 02:34:45.616343 392 mini_ranger.cc:160] Time spent starting Ranger: real 7.800s user 0.004s sys 0.004s
Sep 27, 2026 2:35:05 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 2:35:05 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 27, 2026 2:35:05 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 2:35:05 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 2:35:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 2:35:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 2:35:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 2:35:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 2:35:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 2:35:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 2:35:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 2:35:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 2:35:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 2:35:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 2:35:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 2:35:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 2:35:05 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 2:35:06 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 02:35:06.637017 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 02:35:06 dist-test-slave-grlk krb5kdc[11127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476506, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 02:35:06.732740 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 02:35:06.742053 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 02:35:07.498945 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 02:35:07.499022 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ranger-kms
I20260927 02:35:07.499109 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:35515
2026-09-27 02:35:07,672 [I] DB FLAVOR :POSTGRES
2026-09-27 02:35:07,672 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 02:35:07,673 [I] Checking connection
2026-09-27 02:35:07,673 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcH4HTG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35515/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 02:35:07,911 [I] connection success
2026-09-27 02:35:07,911 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 02:35:07,911 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 02:35:07,911 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcH4HTG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35515/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 02:35:08,157 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 02:35:08,157 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 02:35:08,158 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 02:35:08,158 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcH4HTG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35515/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 02:35:08,407 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 02:35:08.861899 392 mini_ranger_kms.cc:310] Created kms service
I20260927 02:35:09.073132 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 02:35:09.112061 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 02:35:09 dist-test-slave-grlk krb5kdc[11127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476506, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:35:09.477510 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 02:35:09.477654 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskcH4HTG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 02:35:09.477675 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 02:35:09.481262 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11520
I20260927 02:35:09.481349 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:45463
Sep 27, 2026 2:35:10 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 2:35:10 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 2:35:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 2:35:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 2:35:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 2:35:10 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 02:35:11.438932 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.706s user 0.018s sys 0.136s
I20260927 02:35:11.439044 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 02:35:11.439149 392 mini_ranger_kms.cc:401] 127.0.98.20:45463/kms/v1/keys
Sep 27 02:35:11 dist-test-slave-grlk krb5kdc[11127](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790476511, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 02:35:12 dist-test-slave-grlk krb5kdc[11127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476511, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 02:35:12 dist-test-slave-grlk krb5kdc[11127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476511, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 02:35:14 dist-test-slave-grlk krb5kdc[11127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:35:14.501507 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35937
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35937
--ranger_config_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45463/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:35:14.606789 11571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:35:14.607033 11571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:35:14.607069 11571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:35:14.611196 11571 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:35:14.611265 11571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:35:14.611287 11571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:35:14.611307 11571 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:35:14.611342 11571 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:35:14.616714 11571 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:45463/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35937
--ranger_config_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:35:14.617746 11571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:35:14.618721 11571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:35:14.626904 11579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:35:14.626906 11576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:35:14.627089 11571 server_base.cc:1034] running on GCE node
W20260927 02:35:14.626921 11577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:35:14.627856 11571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:35:14.629307 11571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:35:14.630506 11571 hybrid_clock.cc:648] HybridClock initialized: now 1790476514630477 us; error 40 us; skew 500 ppm
Sep 27 02:35:14 dist-test-slave-grlk krb5kdc[11127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:35:14.636703 11571 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:35:14.637578 11571 webserver.cc:460] Webserver started at http://127.0.98.62:34367/ using document root <none> and password file <none>
I20260927 02:35:14.638075 11571 fs_manager.cc:359] Metadata directory not provided
I20260927 02:35:14.638131 11571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:35:14.638321 11571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 02:35:14 dist-test-slave-grlk krb5kdc[11127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:35:14.688053 11571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "a272eb71bbb64c61a0db125a307a63f9"
format_stamp: "Formatted at 2026-09-27 02:35:14 on dist-test-slave-grlk"
server_key: "6cabe4141a7074cae30d76cb4dfac228"
server_key_iv: "e9de639d6665afc7aa66481eb3b4d5e9"
server_key_version: "kuduclusterkey@0"
I20260927 02:35:14.688645 11571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "a272eb71bbb64c61a0db125a307a63f9"
format_stamp: "Formatted at 2026-09-27 02:35:14 on dist-test-slave-grlk"
server_key: "6cabe4141a7074cae30d76cb4dfac228"
server_key_iv: "e9de639d6665afc7aa66481eb3b4d5e9"
server_key_version: "kuduclusterkey@0"
I20260927 02:35:14.692291 11571 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 02:35:14.709774 11586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:35:14.711042 11571 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 02:35:14.711144 11571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "a272eb71bbb64c61a0db125a307a63f9"
format_stamp: "Formatted at 2026-09-27 02:35:14 on dist-test-slave-grlk"
server_key: "6cabe4141a7074cae30d76cb4dfac228"
server_key_iv: "e9de639d6665afc7aa66481eb3b4d5e9"
server_key_version: "kuduclusterkey@0"
I20260927 02:35:14.711269 11571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:35:14.732512 11571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:35:14.734966 11571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:35:14.735126 11571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:35:14.741650 11571 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35937
I20260927 02:35:14.741727 11638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35937 every 8 connection(s)
I20260927 02:35:14.742548 11571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:35:14.745222 11639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:35:14.749172 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 11571
I20260927 02:35:14.749285 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:35:14.749516 392 external_mini_cluster.cc:1428] Setting key 4681ce3e305a5ee0c9275ce167d0e802
I20260927 02:35:14.751255 11639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a272eb71bbb64c61a0db125a307a63f9: Bootstrap starting.
I20260927 02:35:14.753310 11639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a272eb71bbb64c61a0db125a307a63f9: Neither blocks nor log segments found. Creating new log.
I20260927 02:35:14.753990 11639 log.cc:824] T 00000000000000000000000000000000 P a272eb71bbb64c61a0db125a307a63f9: Log is configured to *not* fsync() on all Append() calls
I20260927 02:35:14.755667 11639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a272eb71bbb64c61a0db125a307a63f9: No bootstrap required, opened a new log
Sep 27 02:35:14 dist-test-slave-grlk krb5kdc[11127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:35:14.759600 11639 raft_consensus.cc:348] T 00000000000000000000000000000000 P a272eb71bbb64c61a0db125a307a63f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a272eb71bbb64c61a0db125a307a63f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35937 } }
I20260927 02:35:14.759804 11639 raft_consensus.cc:374] T 00000000000000000000000000000000 P a272eb71bbb64c61a0db125a307a63f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:35:14.759879 11639 raft_consensus.cc:729] T 00000000000000000000000000000000 P a272eb71bbb64c61a0db125a307a63f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a272eb71bbb64c61a0db125a307a63f9, State: Initialized, Role: FOLLOWER
I20260927 02:35:14.760334 11639 consensus_queue.cc:260] T 00000000000000000000000000000000 P a272eb71bbb64c61a0db125a307a63f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a272eb71bbb64c61a0db125a307a63f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35937 } }
I20260927 02:35:14.760468 11639 raft_consensus.cc:388] T 00000000000000000000000000000000 P a272eb71bbb64c61a0db125a307a63f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:35:14.760529 11639 raft_consensus.cc:482] T 00000000000000000000000000000000 P a272eb71bbb64c61a0db125a307a63f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:35:14.760617 11639 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a272eb71bbb64c61a0db125a307a63f9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:35:14.761528 11639 raft_consensus.cc:504] T 00000000000000000000000000000000 P a272eb71bbb64c61a0db125a307a63f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a272eb71bbb64c61a0db125a307a63f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35937 } }
I20260927 02:35:14.761840 11639 leader_election.cc:302] T 00000000000000000000000000000000 P a272eb71bbb64c61a0db125a307a63f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a272eb71bbb64c61a0db125a307a63f9; no voters:
I20260927 02:35:14.762019 11642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:35:14.750638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41426 (local address 127.0.98.62:35937)
0927 02:35:14.750976 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0927 02:35:14.750989 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 02:35:14.751021 (+ 32us) server_negotiation.cc:407] Connection header received
0927 02:35:14.751618 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:35:14.751635 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:35:14.751946 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:35:14.752222 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:35:14.752953 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:35:14.753934 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:35:14.754898 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:35:14.755332 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:35:14.755469 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:35:14.757878 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:35:14.757882 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:35:14.757893 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:35:14.757917 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:35:14.759436 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:35:14.759801 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:35:14.759805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:35:14.759809 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:35:14.759931 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:35:14.760236 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:35:14.760238 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:35:14.760239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:35:14.760612 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:35:14.760779 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 02:35:14.761082 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 02:35:14.761300 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":98,"threads_started":1}
I20260927 02:35:14.762115 11639 leader_election.cc:288] T 00000000000000000000000000000000 P a272eb71bbb64c61a0db125a307a63f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:35:14.762194 11644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a272eb71bbb64c61a0db125a307a63f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:35:14.763311 11639 sys_catalog.cc:564] T 00000000000000000000000000000000 P a272eb71bbb64c61a0db125a307a63f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:35:14.763429 11644 raft_consensus.cc:686] T 00000000000000000000000000000000 P a272eb71bbb64c61a0db125a307a63f9 [term 1 LEADER]: Becoming Leader. State: Replica: a272eb71bbb64c61a0db125a307a63f9, State: Running, Role: LEADER
I20260927 02:35:14.763688 11644 consensus_queue.cc:237] T 00000000000000000000000000000000 P a272eb71bbb64c61a0db125a307a63f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a272eb71bbb64c61a0db125a307a63f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35937 } }
I20260927 02:35:14.764708 11644 sys_catalog.cc:455] T 00000000000000000000000000000000 P a272eb71bbb64c61a0db125a307a63f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a272eb71bbb64c61a0db125a307a63f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a272eb71bbb64c61a0db125a307a63f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35937 } } }
I20260927 02:35:14.764813 11644 sys_catalog.cc:458] T 00000000000000000000000000000000 P a272eb71bbb64c61a0db125a307a63f9 [sys.catalog]: This master's current role is: LEADER
I20260927 02:35:14.765017 11644 sys_catalog.cc:455] T 00000000000000000000000000000000 P a272eb71bbb64c61a0db125a307a63f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a272eb71bbb64c61a0db125a307a63f9. Latest consensus state: current_term: 1 leader_uuid: "a272eb71bbb64c61a0db125a307a63f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a272eb71bbb64c61a0db125a307a63f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35937 } } }
I20260927 02:35:14.765101 11644 sys_catalog.cc:458] T 00000000000000000000000000000000 P a272eb71bbb64c61a0db125a307a63f9 [sys.catalog]: This master's current role is: LEADER
I20260927 02:35:14.765367 11639 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 02:35:14.766319 11657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:35:14.771004 11657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:35:14.776960 11657 catalog_manager.cc:1347] Generated new cluster ID: 1b912cfe9c70424e92e8c4428c301a6c
I20260927 02:35:14.777053 11657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:35:14.785251 11657 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:35:14.786540 11657 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:35:14.794399 11657 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a272eb71bbb64c61a0db125a307a63f9: Generated new TSK 0
I20260927 02:35:14.795217 11657 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:35:15.436167 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45463/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35937
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:35:15.567924 11684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:35:15.568143 11684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:35:15.568198 11684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:35:15.573475 11684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:35:15.573544 11684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:35:15.573625 11684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:35:15.580538 11684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:45463/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:35:15.581523 11684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:35:15.582435 11684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:35:15.593895 11694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:35:15.594086 11691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:35:15.593902 11692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:35:15.595113 11684 server_base.cc:1034] running on GCE node
I20260927 02:35:15.595557 11684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:35:15.596271 11684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:35:15.597455 11684 hybrid_clock.cc:648] HybridClock initialized: now 1790476515597431 us; error 39 us; skew 500 ppm
Sep 27 02:35:15 dist-test-slave-grlk krb5kdc[11127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:35:15.600517 11684 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:35:15.601593 11684 webserver.cc:460] Webserver started at http://127.0.98.1:32775/ using document root <none> and password file <none>
I20260927 02:35:15.602259 11684 fs_manager.cc:359] Metadata directory not provided
I20260927 02:35:15.602344 11684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:35:15.602579 11684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 02:35:15 dist-test-slave-grlk krb5kdc[11127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:35:15.621239 11684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-0/data/instance:
uuid: "5f76221d428e48e0bfce030118cb886c"
format_stamp: "Formatted at 2026-09-27 02:35:15 on dist-test-slave-grlk"
server_key: "062b7329507c71ed1ba10eb1b81e8f6a"
server_key_iv: "1d645fe63b2a87e0487f1e300b862c3c"
server_key_version: "kuduclusterkey@0"
I20260927 02:35:15.623515 11684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5f76221d428e48e0bfce030118cb886c"
format_stamp: "Formatted at 2026-09-27 02:35:15 on dist-test-slave-grlk"
server_key: "062b7329507c71ed1ba10eb1b81e8f6a"
server_key_iv: "1d645fe63b2a87e0487f1e300b862c3c"
server_key_version: "kuduclusterkey@0"
I20260927 02:35:15.632412 11684 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.000s sys 0.003s
I20260927 02:35:15.656271 11703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:35:15.657598 11684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 02:35:15.657729 11684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-0/wal
uuid: "5f76221d428e48e0bfce030118cb886c"
format_stamp: "Formatted at 2026-09-27 02:35:15 on dist-test-slave-grlk"
server_key: "062b7329507c71ed1ba10eb1b81e8f6a"
server_key_iv: "1d645fe63b2a87e0487f1e300b862c3c"
server_key_version: "kuduclusterkey@0"
I20260927 02:35:15.657897 11684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:35:15.699178 11684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:35:15.702675 11684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:35:15.702919 11684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:35:15.703593 11684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:35:15.704593 11684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:35:15.704691 11684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:35:15.704810 11684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:35:15.704885 11684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:35:15.717084 11684 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34697
I20260927 02:35:15.718307 11684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:35:15.719515 11816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34697 every 8 connection(s)
I20260927 02:35:15.723577 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 11684
I20260927 02:35:15.723701 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:35:15.723959 392 external_mini_cluster.cc:1428] Setting key 2c0159037a565bc7318b249b9234a540
Sep 27 02:35:15 dist-test-slave-grlk krb5kdc[11127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:35:15.732563 11819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:35:15.721006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36897 (local address 127.0.98.62:35937)
0927 02:35:15.721280 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0927 02:35:15.721288 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:35:15.722029 (+ 741us) server_negotiation.cc:407] Connection header received
0927 02:35:15.722939 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:35:15.722942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:35:15.722998 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:35:15.723078 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:35:15.724142 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:35:15.724858 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:35:15.726140 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:35:15.726391 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:35:15.726481 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:35:15.729087 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:35:15.729090 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:35:15.729093 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:35:15.729118 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:35:15.730600 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:35:15.731121 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:35:15.731124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:35:15.731126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:35:15.731207 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:35:15.731517 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:35:15.731521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:35:15.731523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:35:15.731701 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:35:15.731781 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 02:35:15.732269 (+ 488us) server_negotiation.cc:299] Negotiation successful
0927 02:35:15.732388 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":98,"threads_started":1}
I20260927 02:35:15.733397 11820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:35:15.721447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35937 (local address 127.0.98.1:36897)
0927 02:35:15.721891 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0927 02:35:15.721939 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 02:35:15.722750 (+ 811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:35:15.723250 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:35:15.723256 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:35:15.723501 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:35:15.724010 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:35:15.724027 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:35:15.725018 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:35:15.725022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:35:15.726026 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:35:15.726035 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:35:15.726523 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:35:15.726527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:35:15.726676 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:35:15.727227 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:35:15.727248 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:35:15.728915 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:35:15.730701 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:35:15.730707 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:35:15.730721 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:35:15.731033 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:35:15.731285 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:35:15.731289 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:35:15.731291 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:35:15.731426 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:35:15.731792 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:35:15.731799 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:35:15.732045 (+ 246us) client_negotiation.cc:769] Sending connection context
0927 02:35:15.732206 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 02:35:15.732496 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":103,"threads_started":1}
I20260927 02:35:15.734477 11817 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35937
I20260927 02:35:15.734699 11817 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:35:15.735148 11817 heartbeater.cc:511] Master 127.0.98.62:35937 requested a full tablet report, sending...
I20260927 02:35:15.736544 11603 ts_manager.cc:194] Registered new tserver with Master: 5f76221d428e48e0bfce030118cb886c (127.0.98.1:34697)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 02:35:15.737943 11603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36897
Sep 27 02:35:15 dist-test-slave-grlk krb5kdc[11127](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790476515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:35:15.759936 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45463/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35937
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:35:15.881069 11823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:35:15.881315 11823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:35:15.881374 11823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:35:15.885455 11823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:35:15.885517 11823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:35:15.885599 11823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:35:15.891462 11823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:45463/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:35:15.892607 11823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:35:15.893589 11823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:35:15.903007 11832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:35:15.903007 11830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:35:15.903447 11829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:35:15.903779 11823 server_base.cc:1034] running on GCE node
I20260927 02:35:15.904152 11823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:35:15.904750 11823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:35:15.905936 11823 hybrid_clock.cc:648] HybridClock initialized: now 1790476515905906 us; error 41 us; skew 500 ppm
Sep 27 02:35:15 dist-test-slave-grlk krb5kdc[11127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:35:15.909368 11823 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:35:15.910627 11823 webserver.cc:460] Webserver started at http://127.0.98.2:39615/ using document root <none> and password file <none>
I20260927 02:35:15.911401 11823 fs_manager.cc:359] Metadata directory not provided
I20260927 02:35:15.911562 11823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:35:15.911931 11823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 02:35:15 dist-test-slave-grlk krb5kdc[11127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:35:15.935398 11823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-1/data/instance:
uuid: "cc031774ac4e463db0a4f3ac3a4ab205"
format_stamp: "Formatted at 2026-09-27 02:35:15 on dist-test-slave-grlk"
server_key: "7202c603e1cea5051012c4430e8dbce3"
server_key_iv: "b29498276b0fa5cf3194041b73ae9f82"
server_key_version: "kuduclusterkey@0"
I20260927 02:35:15.938050 11823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cc031774ac4e463db0a4f3ac3a4ab205"
format_stamp: "Formatted at 2026-09-27 02:35:15 on dist-test-slave-grlk"
server_key: "7202c603e1cea5051012c4430e8dbce3"
server_key_iv: "b29498276b0fa5cf3194041b73ae9f82"
server_key_version: "kuduclusterkey@0"
I20260927 02:35:15.954830 11823 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.004s sys 0.000s
I20260927 02:35:15.977308 11839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:35:15.979038 11823 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 02:35:15.979240 11823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-1/wal
uuid: "cc031774ac4e463db0a4f3ac3a4ab205"
format_stamp: "Formatted at 2026-09-27 02:35:15 on dist-test-slave-grlk"
server_key: "7202c603e1cea5051012c4430e8dbce3"
server_key_iv: "b29498276b0fa5cf3194041b73ae9f82"
server_key_version: "kuduclusterkey@0"
I20260927 02:35:15.979481 11823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:35:16.003346 11823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:35:16.007086 11823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:35:16.007324 11823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:35:16.008033 11823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:35:16.009114 11823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:35:16.009217 11823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:35:16.009310 11823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:35:16.009385 11823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:35:16.019840 11823 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42113
I20260927 02:35:16.021039 11823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:35:16.027333 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 11823
I20260927 02:35:16.027459 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:35:16.027733 392 external_mini_cluster.cc:1428] Setting key 5828ec29cbe48f2f3a38ee6924a796c9
I20260927 02:35:16.033109 11952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42113 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Sep 27 02:35:16 dist-test-slave-grlk krb5kdc[11127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:35:16.050794 11956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:35:16.035401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35937 (local address 127.0.98.2:49687)
0927 02:35:16.036759 (+ 1358us) negotiation.cc:107] Waiting for socket to connect
0927 02:35:16.036815 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 02:35:16.037911 (+ 1096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:35:16.038689 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:35:16.038698 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:35:16.039053 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:35:16.039709 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:35:16.039730 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:35:16.041077 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:35:16.041080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:35:16.042379 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:35:16.042389 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:35:16.042908 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:35:16.042912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:35:16.043099 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:35:16.043773 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:35:16.043794 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:35:16.045659 (+ 1865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:35:16.047778 (+ 2119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:35:16.047784 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:35:16.047802 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:35:16.048113 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:35:16.048434 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:35:16.048437 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:35:16.048439 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:35:16.048608 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:35:16.049124 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:35:16.049132 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:35:16.049506 (+ 374us) client_negotiation.cc:769] Sending connection context
0927 02:35:16.049723 (+ 217us) client_negotiation.cc:240] Negotiation successful
0927 02:35:16.050032 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1139,"thread_start_us":155,"threads_started":1}
I20260927 02:35:16.051509 11819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:35:16.036469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49687 (local address 127.0.98.62:35937)
0927 02:35:16.036612 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 02:35:16.036617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:35:16.036905 (+ 288us) server_negotiation.cc:407] Connection header received
0927 02:35:16.038156 (+ 1251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:35:16.038159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:35:16.038215 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:35:16.038268 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:35:16.039892 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:35:16.040885 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:35:16.042491 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:35:16.042767 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:35:16.042885 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:35:16.045867 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:35:16.045870 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:35:16.045873 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:35:16.045904 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:35:16.047667 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:35:16.048236 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:35:16.048251 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:35:16.048253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:35:16.048339 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:35:16.048708 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:35:16.048711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:35:16.048713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:35:16.048973 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:35:16.049063 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 02:35:16.051206 (+ 2143us) server_negotiation.cc:299] Negotiation successful
0927 02:35:16.051350 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 02:35:16.052862 11953 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35937
I20260927 02:35:16.053133 11953 heartbeater.cc:464] Registering TS with master...
I20260927 02:35:16.053735 11953 heartbeater.cc:511] Master 127.0.98.62:35937 requested a full tablet report, sending...
I20260927 02:35:16.055032 11603 ts_manager.cc:194] Registered new tserver with Master: cc031774ac4e463db0a4f3ac3a4ab205 (127.0.98.2:42113)
I20260927 02:35:16.055804 11603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49687
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:35:16.064792 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45463/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35937
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:35:16.215261 11958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:35:16.215721 11958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:35:16.215817 11958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
Sep 27 02:35:16 dist-test-slave-grlk krb5kdc[11127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260927 02:35:16.221614 11958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:35:16.221698 11958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:35:16.221807 11958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:35:16.229485 11958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:45463/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:35:16.230810 11958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:35:16.232064 11958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:35:16.244258 11963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:35:16.244413 11958 server_base.cc:1034] running on GCE node
W20260927 02:35:16.244576 11964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:35:16.244263 11966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:35:16.245146 11958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:35:16.245882 11958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:35:16.247038 11958 hybrid_clock.cc:648] HybridClock initialized: now 1790476516246974 us; error 74 us; skew 500 ppm
Sep 27 02:35:16 dist-test-slave-grlk krb5kdc[11127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:35:16.250478 11958 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:35:16.251557 11958 webserver.cc:460] Webserver started at http://127.0.98.3:46103/ using document root <none> and password file <none>
I20260927 02:35:16.252276 11958 fs_manager.cc:359] Metadata directory not provided
I20260927 02:35:16.252365 11958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:35:16.252609 11958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 02:35:16 dist-test-slave-grlk krb5kdc[11127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:35:16.268108 11958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-2/data/instance:
uuid: "e9db1b5be11c4c0abb62867a95ee98ca"
format_stamp: "Formatted at 2026-09-27 02:35:16 on dist-test-slave-grlk"
server_key: "6f839398a28039bac436a7fb0303cf93"
server_key_iv: "8a48d4b443bec31f2b9b6751fb8d70dc"
server_key_version: "kuduclusterkey@0"
I20260927 02:35:16.268879 11958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e9db1b5be11c4c0abb62867a95ee98ca"
format_stamp: "Formatted at 2026-09-27 02:35:16 on dist-test-slave-grlk"
server_key: "6f839398a28039bac436a7fb0303cf93"
server_key_iv: "8a48d4b443bec31f2b9b6751fb8d70dc"
server_key_version: "kuduclusterkey@0"
I20260927 02:35:16.298650 11958 fs_manager.cc:691] Time spent creating directory manager: real 0.029s user 0.003s sys 0.000s
I20260927 02:35:16.332131 11973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:35:16.350385 11958 fs_manager.cc:725] Time spent opening block manager: real 0.025s user 0.002s sys 0.000s
I20260927 02:35:16.350553 11958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-2/wal
uuid: "e9db1b5be11c4c0abb62867a95ee98ca"
format_stamp: "Formatted at 2026-09-27 02:35:16 on dist-test-slave-grlk"
server_key: "6f839398a28039bac436a7fb0303cf93"
server_key_iv: "8a48d4b443bec31f2b9b6751fb8d70dc"
server_key_version: "kuduclusterkey@0"
I20260927 02:35:16.350708 11958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:35:16.380389 11958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:35:16.383929 11958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:35:16.384104 11958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:35:16.385115 11958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:35:16.386049 11958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:35:16.386104 11958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:35:16.386157 11958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:35:16.386183 11958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:35:16.395479 11958 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36409
I20260927 02:35:16.396476 11958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:35:16.397759 12086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36409 every 8 connection(s)
I20260927 02:35:16.402257 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 11958
I20260927 02:35:16.402372 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:35:16.402623 392 external_mini_cluster.cc:1428] Setting key 45a9b9b288aa1390ee1c8dd12929e5b9
Sep 27 02:35:16 dist-test-slave-grlk krb5kdc[11127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:35:16.419260 11819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:35:16.401426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40037 (local address 127.0.98.62:35937)
0927 02:35:16.401543 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 02:35:16.401547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:35:16.405416 (+ 3869us) server_negotiation.cc:407] Connection header received
0927 02:35:16.406517 (+ 1101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:35:16.406520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:35:16.406569 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:35:16.406617 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:35:16.408329 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:35:16.409349 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:35:16.411064 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:35:16.411388 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:35:16.411452 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:35:16.414489 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:35:16.414493 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:35:16.414496 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:35:16.414541 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:35:16.417123 (+ 2582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:35:16.417677 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:35:16.417681 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:35:16.417683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:35:16.417762 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:35:16.418095 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:35:16.418098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:35:16.418100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:35:16.418299 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:35:16.418360 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 02:35:16.418982 (+ 622us) server_negotiation.cc:299] Negotiation successful
0927 02:35:16.419096 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 02:35:16.420944 12089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:35:16.400375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35937 (local address 127.0.98.3:40037)
0927 02:35:16.405272 (+ 4897us) negotiation.cc:107] Waiting for socket to connect
0927 02:35:16.405336 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 02:35:16.406298 (+ 962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:35:16.407005 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:35:16.407014 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:35:16.407401 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:35:16.408132 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:35:16.408157 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:35:16.409491 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:35:16.409494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:35:16.410937 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:35:16.410948 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:35:16.411630 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:35:16.411634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:35:16.411808 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:35:16.412459 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:35:16.412479 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:35:16.414316 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:35:16.417260 (+ 2944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:35:16.417265 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:35:16.417280 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:35:16.417576 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:35:16.417856 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:35:16.417859 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:35:16.417861 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:35:16.418011 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:35:16.418431 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:35:16.418438 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:35:16.418735 (+ 297us) client_negotiation.cc:769] Sending connection context
0927 02:35:16.419653 (+ 918us) client_negotiation.cc:240] Negotiation successful
0927 02:35:16.419958 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4687,"thread_start_us":131,"threads_started":1}
I20260927 02:35:16.422322 12087 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35937
I20260927 02:35:16.422531 12087 heartbeater.cc:464] Registering TS with master...
I20260927 02:35:16.423017 12087 heartbeater.cc:511] Master 127.0.98.62:35937 requested a full tablet report, sending...
I20260927 02:35:16.424045 11603 ts_manager.cc:194] Registered new tserver with Master: e9db1b5be11c4c0abb62867a95ee98ca (127.0.98.3:36409)
I20260927 02:35:16.424748 11603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40037
I20260927 02:35:16.428012 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:35:16.439000 11819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:35:16.430022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41438 (local address 127.0.98.62:35937)
0927 02:35:16.430132 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 02:35:16.430136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:35:16.430407 (+ 271us) server_negotiation.cc:407] Connection header received
0927 02:35:16.430519 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:35:16.430523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:35:16.430567 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:35:16.430680 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:35:16.431466 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:35:16.432288 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:35:16.433499 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:35:16.433837 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:35:16.433957 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:35:16.435551 (+ 1594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:35:16.435555 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:35:16.435558 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:35:16.435583 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:35:16.437323 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:35:16.437783 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:35:16.437786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:35:16.437788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:35:16.437874 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:35:16.438153 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:35:16.438156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:35:16.438158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:35:16.438362 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:35:16.438484 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 02:35:16.438740 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 02:35:16.438852 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 02:35:16.442974 11603 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41438:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:35:16.446951 11603 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:35:16.465767 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:35:16.456601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33778 (local address 127.0.98.1:34697)
0927 02:35:16.457112 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0927 02:35:16.457120 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:35:16.458569 (+ 1449us) server_negotiation.cc:407] Connection header received
0927 02:35:16.459039 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:35:16.459045 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:35:16.459234 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:35:16.459348 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:35:16.460444 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:35:16.461619 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:35:16.464019 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:35:16.464831 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:35:16.464948 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:35:16.465097 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 02:35:16.465511 (+ 414us) server_negotiation.cc:299] Negotiation successful
0927 02:35:16.465610 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":102,"threads_started":1}
I20260927 02:35:16.467305 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:35:16.456450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34697 (local address 127.0.0.1:33778)
0927 02:35:16.458431 (+ 1981us) negotiation.cc:107] Waiting for socket to connect
0927 02:35:16.458480 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 02:35:16.458880 (+ 400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:35:16.460223 (+ 1343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:35:16.460226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:35:16.460243 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:35:16.460327 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:35:16.460333 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:35:16.461758 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:35:16.461761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:35:16.463878 (+ 2117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:35:16.463889 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:35:16.465211 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:35:16.465214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:35:16.465311 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:35:16.465346 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 02:35:16.466196 (+ 850us) client_negotiation.cc:240] Negotiation successful
0927 02:35:16.466322 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1817,"thread_start_us":88,"threads_started":1}
I20260927 02:35:16.469040 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:35:16.457657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36409 (local address 127.0.0.1:53788)
0927 02:35:16.458712 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0927 02:35:16.458725 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:35:16.458842 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:35:16.459707 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:35:16.459713 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:35:16.459735 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:35:16.459850 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:35:16.459856 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:35:16.464776 (+ 4920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:35:16.464779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:35:16.465895 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:35:16.465906 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:35:16.467245 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:35:16.467249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:35:16.467337 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:35:16.467359 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 02:35:16.467424 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 02:35:16.467487 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1002,"thread_start_us":74,"threads_started":1}
I20260927 02:35:16.469415 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:35:16.457866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42113 (local address 127.0.0.1:57796)
0927 02:35:16.461072 (+ 3206us) negotiation.cc:107] Waiting for socket to connect
0927 02:35:16.461086 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:35:16.461170 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:35:16.461823 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:35:16.461825 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:35:16.461842 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:35:16.461923 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:35:16.461928 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:35:16.463237 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:35:16.463241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:35:16.464344 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:35:16.464354 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:35:16.468823 (+ 4469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:35:16.468827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:35:16.468964 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:35:16.468988 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 02:35:16.469059 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 02:35:16.469294 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3146,"thread_start_us":76,"threads_started":1}
I20260927 02:35:16.469544 12103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:35:16.458005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53788 (local address 127.0.98.3:36409)
0927 02:35:16.458762 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0927 02:35:16.458773 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:35:16.458795 (+ 22us) server_negotiation.cc:407] Connection header received
0927 02:35:16.459018 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:35:16.459024 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:35:16.459230 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:35:16.459366 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:35:16.459973 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:35:16.464664 (+ 4691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:35:16.466050 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:35:16.466941 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:35:16.467042 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:35:16.467173 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 02:35:16.469312 (+ 2139us) server_negotiation.cc:299] Negotiation successful
0927 02:35:16.469409 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":97,"threads_started":1}
I20260927 02:35:16.470604 12104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:35:16.458246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57796 (local address 127.0.98.2:42113)
0927 02:35:16.459628 (+ 1382us) server_negotiation.cc:206] Beginning negotiation
0927 02:35:16.459641 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 02:35:16.461273 (+ 1632us) server_negotiation.cc:407] Connection header received
0927 02:35:16.461349 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:35:16.461353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:35:16.461542 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:35:16.461686 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:35:16.462042 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:35:16.463107 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:35:16.467833 (+ 4726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:35:16.468693 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:35:16.468802 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:35:16.470324 (+ 1522us) server_negotiation.cc:1035] Waiting for connection context
0927 02:35:16.470380 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 02:35:16.470473 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1287,"thread_start_us":96,"threads_started":1}
I20260927 02:35:16.470796 11751 tablet_service.cc:1467] Processing CreateTablet for tablet a2ebf63298884f3a915bb2bb072c6e76 (DEFAULT_TABLE table=test-table [id=ea9119cffada419fb046641a3f35f0dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:35:16.471920 11751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2ebf63298884f3a915bb2bb072c6e76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:35:16.472172 12021 tablet_service.cc:1467] Processing CreateTablet for tablet a2ebf63298884f3a915bb2bb072c6e76 (DEFAULT_TABLE table=test-table [id=ea9119cffada419fb046641a3f35f0dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:35:16.473157 12021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2ebf63298884f3a915bb2bb072c6e76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:35:16.473017 11887 tablet_service.cc:1467] Processing CreateTablet for tablet a2ebf63298884f3a915bb2bb072c6e76 (DEFAULT_TABLE table=test-table [id=ea9119cffada419fb046641a3f35f0dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:35:16.473865 11887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2ebf63298884f3a915bb2bb072c6e76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:35:16.481261 12106 tablet_bootstrap.cc:492] T a2ebf63298884f3a915bb2bb072c6e76 P e9db1b5be11c4c0abb62867a95ee98ca: Bootstrap starting.
I20260927 02:35:16.482612 12107 tablet_bootstrap.cc:492] T a2ebf63298884f3a915bb2bb072c6e76 P cc031774ac4e463db0a4f3ac3a4ab205: Bootstrap starting.
I20260927 02:35:16.484174 12106 tablet_bootstrap.cc:654] T a2ebf63298884f3a915bb2bb072c6e76 P e9db1b5be11c4c0abb62867a95ee98ca: Neither blocks nor log segments found. Creating new log.
I20260927 02:35:16.485117 12106 log.cc:824] T a2ebf63298884f3a915bb2bb072c6e76 P e9db1b5be11c4c0abb62867a95ee98ca: Log is configured to *not* fsync() on all Append() calls
I20260927 02:35:16.485229 12107 tablet_bootstrap.cc:654] T a2ebf63298884f3a915bb2bb072c6e76 P cc031774ac4e463db0a4f3ac3a4ab205: Neither blocks nor log segments found. Creating new log.
I20260927 02:35:16.486030 12107 log.cc:824] T a2ebf63298884f3a915bb2bb072c6e76 P cc031774ac4e463db0a4f3ac3a4ab205: Log is configured to *not* fsync() on all Append() calls
I20260927 02:35:16.487313 12106 tablet_bootstrap.cc:492] T a2ebf63298884f3a915bb2bb072c6e76 P e9db1b5be11c4c0abb62867a95ee98ca: No bootstrap required, opened a new log
I20260927 02:35:16.487496 12106 ts_tablet_manager.cc:1397] T a2ebf63298884f3a915bb2bb072c6e76 P e9db1b5be11c4c0abb62867a95ee98ca: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 02:35:16.487740 12107 tablet_bootstrap.cc:492] T a2ebf63298884f3a915bb2bb072c6e76 P cc031774ac4e463db0a4f3ac3a4ab205: No bootstrap required, opened a new log
I20260927 02:35:16.487916 12107 ts_tablet_manager.cc:1397] T a2ebf63298884f3a915bb2bb072c6e76 P cc031774ac4e463db0a4f3ac3a4ab205: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 02:35:16.491829 12106 raft_consensus.cc:348] T a2ebf63298884f3a915bb2bb072c6e76 P e9db1b5be11c4c0abb62867a95ee98ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f76221d428e48e0bfce030118cb886c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34697 } } peers { permanent_uuid: "e9db1b5be11c4c0abb62867a95ee98ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36409 } } peers { permanent_uuid: "cc031774ac4e463db0a4f3ac3a4ab205" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42113 } }
I20260927 02:35:16.492079 12107 raft_consensus.cc:348] T a2ebf63298884f3a915bb2bb072c6e76 P cc031774ac4e463db0a4f3ac3a4ab205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f76221d428e48e0bfce030118cb886c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34697 } } peers { permanent_uuid: "e9db1b5be11c4c0abb62867a95ee98ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36409 } } peers { permanent_uuid: "cc031774ac4e463db0a4f3ac3a4ab205" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42113 } }
I20260927 02:35:16.492250 12106 raft_consensus.cc:374] T a2ebf63298884f3a915bb2bb072c6e76 P e9db1b5be11c4c0abb62867a95ee98ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:35:16.492331 12107 raft_consensus.cc:374] T a2ebf63298884f3a915bb2bb072c6e76 P cc031774ac4e463db0a4f3ac3a4ab205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:35:16.492388 12107 raft_consensus.cc:729] T a2ebf63298884f3a915bb2bb072c6e76 P cc031774ac4e463db0a4f3ac3a4ab205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc031774ac4e463db0a4f3ac3a4ab205, State: Initialized, Role: FOLLOWER
I20260927 02:35:16.492414 12106 raft_consensus.cc:729] T a2ebf63298884f3a915bb2bb072c6e76 P e9db1b5be11c4c0abb62867a95ee98ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9db1b5be11c4c0abb62867a95ee98ca, State: Initialized, Role: FOLLOWER
I20260927 02:35:16.493010 12106 consensus_queue.cc:260] T a2ebf63298884f3a915bb2bb072c6e76 P e9db1b5be11c4c0abb62867a95ee98ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f76221d428e48e0bfce030118cb886c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34697 } } peers { permanent_uuid: "e9db1b5be11c4c0abb62867a95ee98ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36409 } } peers { permanent_uuid: "cc031774ac4e463db0a4f3ac3a4ab205" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42113 } }
I20260927 02:35:16.493882 12107 consensus_queue.cc:260] T a2ebf63298884f3a915bb2bb072c6e76 P cc031774ac4e463db0a4f3ac3a4ab205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f76221d428e48e0bfce030118cb886c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34697 } } peers { permanent_uuid: "e9db1b5be11c4c0abb62867a95ee98ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36409 } } peers { permanent_uuid: "cc031774ac4e463db0a4f3ac3a4ab205" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42113 } }
I20260927 02:35:16.494346 12106 ts_tablet_manager.cc:1428] T a2ebf63298884f3a915bb2bb072c6e76 P e9db1b5be11c4c0abb62867a95ee98ca: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260927 02:35:16.494737 12087 heartbeater.cc:503] Master 127.0.98.62:35937 was elected leader, sending a full tablet report...
I20260927 02:35:16.494776 12107 ts_tablet_manager.cc:1428] T a2ebf63298884f3a915bb2bb072c6e76 P cc031774ac4e463db0a4f3ac3a4ab205: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260927 02:35:16.495776 11953 heartbeater.cc:503] Master 127.0.98.62:35937 was elected leader, sending a full tablet report...
I20260927 02:35:16.497310 12105 tablet_bootstrap.cc:492] T a2ebf63298884f3a915bb2bb072c6e76 P 5f76221d428e48e0bfce030118cb886c: Bootstrap starting.
I20260927 02:35:16.499524 12105 tablet_bootstrap.cc:654] T a2ebf63298884f3a915bb2bb072c6e76 P 5f76221d428e48e0bfce030118cb886c: Neither blocks nor log segments found. Creating new log.
I20260927 02:35:16.500252 12105 log.cc:824] T a2ebf63298884f3a915bb2bb072c6e76 P 5f76221d428e48e0bfce030118cb886c: Log is configured to *not* fsync() on all Append() calls
I20260927 02:35:16.502276 12105 tablet_bootstrap.cc:492] T a2ebf63298884f3a915bb2bb072c6e76 P 5f76221d428e48e0bfce030118cb886c: No bootstrap required, opened a new log
I20260927 02:35:16.502455 12105 ts_tablet_manager.cc:1397] T a2ebf63298884f3a915bb2bb072c6e76 P 5f76221d428e48e0bfce030118cb886c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 02:35:16.506574 12105 raft_consensus.cc:348] T a2ebf63298884f3a915bb2bb072c6e76 P 5f76221d428e48e0bfce030118cb886c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f76221d428e48e0bfce030118cb886c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34697 } } peers { permanent_uuid: "e9db1b5be11c4c0abb62867a95ee98ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36409 } } peers { permanent_uuid: "cc031774ac4e463db0a4f3ac3a4ab205" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42113 } }
I20260927 02:35:16.506822 12105 raft_consensus.cc:374] T a2ebf63298884f3a915bb2bb072c6e76 P 5f76221d428e48e0bfce030118cb886c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:35:16.506878 12105 raft_consensus.cc:729] T a2ebf63298884f3a915bb2bb072c6e76 P 5f76221d428e48e0bfce030118cb886c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f76221d428e48e0bfce030118cb886c, State: Initialized, Role: FOLLOWER
I20260927 02:35:16.507324 12105 consensus_queue.cc:260] T a2ebf63298884f3a915bb2bb072c6e76 P 5f76221d428e48e0bfce030118cb886c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f76221d428e48e0bfce030118cb886c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34697 } } peers { permanent_uuid: "e9db1b5be11c4c0abb62867a95ee98ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36409 } } peers { permanent_uuid: "cc031774ac4e463db0a4f3ac3a4ab205" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42113 } }
I20260927 02:35:16.507947 11817 heartbeater.cc:503] Master 127.0.98.62:35937 was elected leader, sending a full tablet report...
I20260927 02:35:16.508252 12105 ts_tablet_manager.cc:1428] T a2ebf63298884f3a915bb2bb072c6e76 P 5f76221d428e48e0bfce030118cb886c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260927 02:35:16.535107 11954 tablet.cc:2369] T a2ebf63298884f3a915bb2bb072c6e76 P cc031774ac4e463db0a4f3ac3a4ab205: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 02:35:16 dist-test-slave-grlk krb5kdc[11127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260927 02:35:16.652386 12088 tablet.cc:2369] T a2ebf63298884f3a915bb2bb072c6e76 P e9db1b5be11c4c0abb62867a95ee98ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:35:16.709964 12113 raft_consensus.cc:482] T a2ebf63298884f3a915bb2bb072c6e76 P 5f76221d428e48e0bfce030118cb886c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:35:16.710227 12113 raft_consensus.cc:504] T a2ebf63298884f3a915bb2bb072c6e76 P 5f76221d428e48e0bfce030118cb886c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f76221d428e48e0bfce030118cb886c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34697 } } peers { permanent_uuid: "e9db1b5be11c4c0abb62867a95ee98ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36409 } } peers { permanent_uuid: "cc031774ac4e463db0a4f3ac3a4ab205" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42113 } }
I20260927 02:35:16.711889 12113 leader_election.cc:288] T a2ebf63298884f3a915bb2bb072c6e76 P 5f76221d428e48e0bfce030118cb886c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9db1b5be11c4c0abb62867a95ee98ca (127.0.98.3:36409), cc031774ac4e463db0a4f3ac3a4ab205 (127.0.98.2:42113)
I20260927 02:35:16.718353 12103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:35:16.713143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44699 (local address 127.0.98.3:36409)
0927 02:35:16.713262 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 02:35:16.713267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:35:16.713682 (+ 415us) server_negotiation.cc:407] Connection header received
0927 02:35:16.713971 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:35:16.713975 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:35:16.714027 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:35:16.714078 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:35:16.714548 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:35:16.715337 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:35:16.717052 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:35:16.717661 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:35:16.717738 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:35:16.717803 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 02:35:16.718146 (+ 343us) server_negotiation.cc:299] Negotiation successful
0927 02:35:16.718221 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 02:35:16.719079 12116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:35:16.712286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36409 (local address 127.0.98.1:44699)
0927 02:35:16.713588 (+ 1302us) negotiation.cc:107] Waiting for socket to connect
0927 02:35:16.713603 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:35:16.713821 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:35:16.714228 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:35:16.714231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:35:16.714264 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:35:16.714419 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:35:16.714427 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:35:16.715546 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:35:16.715549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:35:16.716879 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:35:16.716891 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:35:16.717900 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:35:16.717903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:35:16.718003 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:35:16.718028 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 02:35:16.718843 (+ 815us) client_negotiation.cc:240] Negotiation successful
0927 02:35:16.718924 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1230,"thread_start_us":127,"threads_started":1}
I20260927 02:35:16.719833 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ebf63298884f3a915bb2bb072c6e76" candidate_uuid: "5f76221d428e48e0bfce030118cb886c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9db1b5be11c4c0abb62867a95ee98ca" is_pre_election: true
I20260927 02:35:16.720156 12041 raft_consensus.cc:2454] T a2ebf63298884f3a915bb2bb072c6e76 P e9db1b5be11c4c0abb62867a95ee98ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f76221d428e48e0bfce030118cb886c in term 0.
I20260927 02:35:16.720736 11704 leader_election.cc:302] T a2ebf63298884f3a915bb2bb072c6e76 P 5f76221d428e48e0bfce030118cb886c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f76221d428e48e0bfce030118cb886c, e9db1b5be11c4c0abb62867a95ee98ca; no voters:
I20260927 02:35:16.721052 12113 raft_consensus.cc:2781] T a2ebf63298884f3a915bb2bb072c6e76 P 5f76221d428e48e0bfce030118cb886c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:35:16.721125 12113 raft_consensus.cc:482] T a2ebf63298884f3a915bb2bb072c6e76 P 5f76221d428e48e0bfce030118cb886c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:35:16.721168 12113 raft_consensus.cc:3037] T a2ebf63298884f3a915bb2bb072c6e76 P 5f76221d428e48e0bfce030118cb886c [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:35:16.722434 12113 raft_consensus.cc:504] T a2ebf63298884f3a915bb2bb072c6e76 P 5f76221d428e48e0bfce030118cb886c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f76221d428e48e0bfce030118cb886c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34697 } } peers { permanent_uuid: "e9db1b5be11c4c0abb62867a95ee98ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36409 } } peers { permanent_uuid: "cc031774ac4e463db0a4f3ac3a4ab205" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42113 } }
I20260927 02:35:16.722803 12113 leader_election.cc:288] T a2ebf63298884f3a915bb2bb072c6e76 P 5f76221d428e48e0bfce030118cb886c [CANDIDATE]: Term 1 election: Requested vote from peers e9db1b5be11c4c0abb62867a95ee98ca (127.0.98.3:36409), cc031774ac4e463db0a4f3ac3a4ab205 (127.0.98.2:42113)
I20260927 02:35:16.723392 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ebf63298884f3a915bb2bb072c6e76" candidate_uuid: "5f76221d428e48e0bfce030118cb886c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9db1b5be11c4c0abb62867a95ee98ca"
I20260927 02:35:16.723515 12041 raft_consensus.cc:3037] T a2ebf63298884f3a915bb2bb072c6e76 P e9db1b5be11c4c0abb62867a95ee98ca [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:35:16.724757 12041 raft_consensus.cc:2454] T a2ebf63298884f3a915bb2bb072c6e76 P e9db1b5be11c4c0abb62867a95ee98ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f76221d428e48e0bfce030118cb886c in term 1.
I20260927 02:35:16.725140 11704 leader_election.cc:302] T a2ebf63298884f3a915bb2bb072c6e76 P 5f76221d428e48e0bfce030118cb886c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f76221d428e48e0bfce030118cb886c, e9db1b5be11c4c0abb62867a95ee98ca; no voters:
I20260927 02:35:16.725348 12113 raft_consensus.cc:2781] T a2ebf63298884f3a915bb2bb072c6e76 P 5f76221d428e48e0bfce030118cb886c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:35:16.725565 12113 raft_consensus.cc:686] T a2ebf63298884f3a915bb2bb072c6e76 P 5f76221d428e48e0bfce030118cb886c [term 1 LEADER]: Becoming Leader. State: Replica: 5f76221d428e48e0bfce030118cb886c, State: Running, Role: LEADER
I20260927 02:35:16.725899 12113 consensus_queue.cc:237] T a2ebf63298884f3a915bb2bb072c6e76 P 5f76221d428e48e0bfce030118cb886c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f76221d428e48e0bfce030118cb886c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34697 } } peers { permanent_uuid: "e9db1b5be11c4c0abb62867a95ee98ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36409 } } peers { permanent_uuid: "cc031774ac4e463db0a4f3ac3a4ab205" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42113 } }
I20260927 02:35:16.726512 12104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:35:16.712870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46609 (local address 127.0.98.2:42113)
0927 02:35:16.713018 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 02:35:16.713023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:35:16.716684 (+ 3661us) server_negotiation.cc:407] Connection header received
0927 02:35:16.716881 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:35:16.716885 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:35:16.716948 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:35:16.716996 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:35:16.717362 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:35:16.718139 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:35:16.720103 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:35:16.720691 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:35:16.720772 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:35:16.720826 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 02:35:16.726296 (+ 5470us) server_negotiation.cc:299] Negotiation successful
0927 02:35:16.726392 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 02:35:16.727084 12117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:35:16.712732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42113 (local address 127.0.98.1:46609)
0927 02:35:16.716593 (+ 3861us) negotiation.cc:107] Waiting for socket to connect
0927 02:35:16.716608 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:35:16.716746 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:35:16.717121 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:35:16.717124 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:35:16.717140 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:35:16.717242 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:35:16.717247 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:35:16.718277 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:35:16.718280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:35:16.719470 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:35:16.719480 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:35:16.726035 (+ 6555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:35:16.726039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:35:16.726156 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:35:16.726180 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 02:35:16.726677 (+ 497us) client_negotiation.cc:240] Negotiation successful
0927 02:35:16.726755 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3796,"thread_start_us":103,"threads_started":1}
W20260927 02:35:16.727231 11818 tablet.cc:2369] T a2ebf63298884f3a915bb2bb072c6e76 P 5f76221d428e48e0bfce030118cb886c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:35:16.727936 11907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ebf63298884f3a915bb2bb072c6e76" candidate_uuid: "5f76221d428e48e0bfce030118cb886c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc031774ac4e463db0a4f3ac3a4ab205" is_pre_election: true
I20260927 02:35:16.728197 11907 raft_consensus.cc:2454] T a2ebf63298884f3a915bb2bb072c6e76 P cc031774ac4e463db0a4f3ac3a4ab205 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f76221d428e48e0bfce030118cb886c in term 0.
I20260927 02:35:16.728487 11906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ebf63298884f3a915bb2bb072c6e76" candidate_uuid: "5f76221d428e48e0bfce030118cb886c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc031774ac4e463db0a4f3ac3a4ab205"
I20260927 02:35:16.728586 11906 raft_consensus.cc:3037] T a2ebf63298884f3a915bb2bb072c6e76 P cc031774ac4e463db0a4f3ac3a4ab205 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:35:16.729231 11602 catalog_manager.cc:5495] T a2ebf63298884f3a915bb2bb072c6e76 P 5f76221d428e48e0bfce030118cb886c reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f76221d428e48e0bfce030118cb886c (127.0.98.1). New cstate: current_term: 1 leader_uuid: "5f76221d428e48e0bfce030118cb886c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f76221d428e48e0bfce030118cb886c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9db1b5be11c4c0abb62867a95ee98ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc031774ac4e463db0a4f3ac3a4ab205" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42113 } health_report { overall_health: UNKNOWN } } }
I20260927 02:35:16.729806 11906 raft_consensus.cc:2454] T a2ebf63298884f3a915bb2bb072c6e76 P cc031774ac4e463db0a4f3ac3a4ab205 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f76221d428e48e0bfce030118cb886c in term 1.
I20260927 02:35:16.763449 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:35:16.757665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33792 (local address 127.0.98.1:34697)
0927 02:35:16.757805 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 02:35:16.757809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:35:16.757829 (+ 20us) server_negotiation.cc:407] Connection header received
0927 02:35:16.757917 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:35:16.757921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:35:16.757991 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:35:16.758103 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:35:16.758503 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:35:16.759306 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:35:16.760603 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:35:16.760990 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:35:16.761084 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:35:16.761537 (+ 453us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:35:16.762576 (+ 1039us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:35:16.762697 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 02:35:16.763230 (+ 533us) server_negotiation.cc:299] Negotiation successful
0927 02:35:16.763297 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 02:35:16.770440 12041 raft_consensus.cc:1261] T a2ebf63298884f3a915bb2bb072c6e76 P e9db1b5be11c4c0abb62867a95ee98ca [term 1 FOLLOWER]: Refusing update from remote peer 5f76221d428e48e0bfce030118cb886c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:35:16.770437 11906 raft_consensus.cc:1261] T a2ebf63298884f3a915bb2bb072c6e76 P cc031774ac4e463db0a4f3ac3a4ab205 [term 1 FOLLOWER]: Refusing update from remote peer 5f76221d428e48e0bfce030118cb886c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:35:16.771186 12118 consensus_queue.cc:1035] T a2ebf63298884f3a915bb2bb072c6e76 P 5f76221d428e48e0bfce030118cb886c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9db1b5be11c4c0abb62867a95ee98ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:35:16.771435 12113 consensus_queue.cc:1035] T a2ebf63298884f3a915bb2bb072c6e76 P 5f76221d428e48e0bfce030118cb886c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc031774ac4e463db0a4f3ac3a4ab205" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:35:16.779018 12122 mvcc.cc:205] Tried to move back new op lower bound from 7333791812682661888 to 7333791812513169408. Current Snapshot: MvccSnapshot[applied={T|T < 7333791812682661888}]
I20260927 02:35:16.788868 12124 mvcc.cc:205] Tried to move back new op lower bound from 7333791812682661888 to 7333791812513169408. Current Snapshot: MvccSnapshot[applied={T|T < 7333791812682661888 or (T in {7333791812682661888})}]
I20260927 02:35:16.789382 11602 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:35:16.789593 11602 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:35:16.791899 11602 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a272eb71bbb64c61a0db125a307a63f9: Sending DeleteTablet for 3 replicas of tablet a2ebf63298884f3a915bb2bb072c6e76
I20260927 02:35:16.792483 11751 tablet_service.cc:1514] Processing DeleteTablet for tablet a2ebf63298884f3a915bb2bb072c6e76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:35:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33778
I20260927 02:35:16.792496 12021 tablet_service.cc:1514] Processing DeleteTablet for tablet a2ebf63298884f3a915bb2bb072c6e76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:35:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53788
I20260927 02:35:16.792793 11887 tablet_service.cc:1514] Processing DeleteTablet for tablet a2ebf63298884f3a915bb2bb072c6e76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:35:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57796
I20260927 02:35:16.793793 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 11684
I20260927 02:35:16.794405 12139 tablet_replica.cc:330] T a2ebf63298884f3a915bb2bb072c6e76 P e9db1b5be11c4c0abb62867a95ee98ca: stopping tablet replica
I20260927 02:35:16.794657 12139 raft_consensus.cc:2229] T a2ebf63298884f3a915bb2bb072c6e76 P e9db1b5be11c4c0abb62867a95ee98ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:35:16.794901 12139 raft_consensus.cc:2258] T a2ebf63298884f3a915bb2bb072c6e76 P e9db1b5be11c4c0abb62867a95ee98ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:35:16.795189 12140 tablet_replica.cc:330] T a2ebf63298884f3a915bb2bb072c6e76 P cc031774ac4e463db0a4f3ac3a4ab205: stopping tablet replica
I20260927 02:35:16.795429 12140 raft_consensus.cc:2229] T a2ebf63298884f3a915bb2bb072c6e76 P cc031774ac4e463db0a4f3ac3a4ab205 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:35:16.795650 12140 raft_consensus.cc:2258] T a2ebf63298884f3a915bb2bb072c6e76 P cc031774ac4e463db0a4f3ac3a4ab205 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:35:16.796900 12139 ts_tablet_manager.cc:1905] T a2ebf63298884f3a915bb2bb072c6e76 P e9db1b5be11c4c0abb62867a95ee98ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:35:16.797966 12140 ts_tablet_manager.cc:1905] T a2ebf63298884f3a915bb2bb072c6e76 P cc031774ac4e463db0a4f3ac3a4ab205: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:35:16.799356 12139 ts_tablet_manager.cc:1918] T a2ebf63298884f3a915bb2bb072c6e76 P e9db1b5be11c4c0abb62867a95ee98ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:35:16.799503 12139 log.cc:1196] T a2ebf63298884f3a915bb2bb072c6e76 P e9db1b5be11c4c0abb62867a95ee98ca: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476410933508-392-0/minicluster-data/ts-2/wal/wals/a2ebf63298884f3a915bb2bb072c6e76
W20260927 02:35:16.799533 11590 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34697 (error 108)
I20260927 02:35:16.799917 12139 ts_tablet_manager.cc:1939] T a2ebf63298884f3a915bb2bb072c6e76 P e9db1b5be11c4c0abb62867a95ee98ca: Deleting consensus metadata
I20260927 02:35:16.800211 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 11823
I20260927 02:35:16.801618 11587 catalog_manager.cc:4841] TS e9db1b5be11c4c0abb62867a95ee98ca (127.0.98.3:36409): tablet a2ebf63298884f3a915bb2bb072c6e76 (table test-table [id=ea9119cffada419fb046641a3f35f0dd]) successfully deleted
I20260927 02:35:16.802171 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:35:16.801918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34697 (local address 127.0.0.1:33806)
0927 02:35:16.802034 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 02:35:16.802096 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 02:35:16.802356 11590 catalog_manager.cc:4584] TS 5f76221d428e48e0bfce030118cb886c (127.0.98.1:34697): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a2ebf63298884f3a915bb2bb072c6e76: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34697: connect: Connection refused (error 111)
I20260927 02:35:16.805868 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 11958
I20260927 02:35:16.807265 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:35:16.805975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42113 (local address 127.0.0.1:57810)
0927 02:35:16.807162 (+ 1187us) negotiation.cc:107] Waiting for socket to connect
0927 02:35:16.807206 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1125}
I20260927 02:35:16.810379 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 11571
I20260927 02:35:16.815483 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 02:35:17.149168 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 02:35:17.149277 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 02:35:17.483675 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 02:35:17.483 UTC [11157] LOG: received smart shutdown request
2026-09-27 02:35:17.486 UTC [11157] LOG: background worker "logical replication launcher" (PID 11166) exited with exit code 1
2026-09-27 02:35:17.486 UTC [11161] LOG: shutting down
2026-09-27 02:35:17.644 UTC [11157] LOG: database system is shut down
2026-09-27T02:35:17Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (44166 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:35:17 dist-test-slave-grlk krb5kdc[12150](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:35:17 dist-test-slave-grlk krb5kdc[12150](info): set up 2 sockets
Sep 27 02:35:17 dist-test-slave-grlk krb5kdc[12150](info): commencing operation
krb5kdc: starting...
W20260927 02:35:18.684777 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/test-user.keytab.
Sep 27 02:35:18 dist-test-slave-grlk krb5kdc[12150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:35:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:35:18Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 02:35:18.833801 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskcH4HTG/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/postgres -l logfile start
2026-09-27 02:35:21.097 UTC [12180] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 02:35:21.097 UTC [12180] LOG: listening on IPv4 address "127.0.98.20", port 41113
2026-09-27 02:35:21.100 UTC [12180] LOG: listening on Unix socket "/tmp/.s.PGSQL.41113"
2026-09-27 02:35:21.126 UTC [12183] LOG: database system was shut down at 2026-09-27 02:35:19 UTC
2026-09-27 02:35:21.129 UTC [12180] LOG: database system is ready to accept connections
I20260927 02:35:21.992508 392 mini_postgres.cc:96] Postgres bound to 41113
2026-09-27 02:35:21.998 UTC [12191] FATAL: database "slave" does not exist
127.0.98.20:41113 - accepting connections
I20260927 02:35:21.998461 392 mini_ranger.cc:161] Starting Ranger...
I20260927 02:35:22.011593 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 02:35:22.404866 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 02:35:22.404968 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ranger-admin
2026-09-27 02:35:22,661 [I] DB FLAVOR :POSTGRES
2026-09-27 02:35:22,662 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 02:35:22,662 [I] Checking connection..
2026-09-27 02:35:22,662 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcH4HTG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41113/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 02:35:22,911 [I] Checking connection passed.
2026-09-27 02:35:22,911 [I] --------- Verifying version history table ---------
2026-09-27 02:35:22,911 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcH4HTG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41113/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 02:35:23,164 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 02:35:23,164 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcH4HTG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41113/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 02:35:23,408 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 02:35:23,409 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 02:35:23,409 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcH4HTG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41113/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 02:35:23.652 UTC [12255] WARNING: there is no transaction in progress
2026-09-27 02:35:23,657 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 02:35:23,657 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcH4HTG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41113/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 02:35:23,892 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 02:35:23,892 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 02:35:23,892 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcH4HTG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41113/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 02:35:24,104 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcH4HTG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41113/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 02:35:24,324 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcH4HTG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41113/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-grlk.c.gcp-upstream.internal','N') ;"
2026-09-27 02:35:24,543 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcH4HTG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41113/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 02:35:24,777 [I] Table x_portal_user does not exist in database ranger
2026-09-27 02:35:24,778 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcH4HTG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41113/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 02:35:24,998 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 02:35:24,998 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcH4HTG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41113/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 02:35:25,230 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcH4HTG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41113/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 02:35:25,472 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 02:35:25,472 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcH4HTG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41113/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 02:35:26.647 UTC [12363] WARNING: there is no transaction in progress
2026-09-27 02:35:26.661 UTC [12363] WARNING: there is no transaction in progress
2026-09-27 02:35:26.676 UTC [12363] WARNING: there is no transaction in progress
2026-09-27 02:35:26.691 UTC [12363] WARNING: there is no transaction in progress
2026-09-27 02:35:26.706 UTC [12363] WARNING: there is no transaction in progress
2026-09-27 02:35:26.722 UTC [12363] WARNING: there is no transaction in progress
2026-09-27 02:35:26.774 UTC [12363] WARNING: there is no transaction in progress
2026-09-27 02:35:26.780 UTC [12363] WARNING: there is no transaction in progress
2026-09-27 02:35:26.794 UTC [12363] WARNING: there is no transaction in progress
2026-09-27 02:35:26.803 UTC [12363] WARNING: there is no transaction in progress
2026-09-27 02:35:26.813 UTC [12363] WARNING: there is no transaction in progress
2026-09-27 02:35:26.826 UTC [12363] WARNING: there is no transaction in progress
2026-09-27 02:35:26.836 UTC [12363] WARNING: there is no transaction in progress
2026-09-27 02:35:26.842 UTC [12363] WARNING: there is no transaction in progress
2026-09-27 02:35:26.848 UTC [12363] WARNING: there is no transaction in progress
2026-09-27 02:35:26.854 UTC [12363] WARNING: there is no transaction in progress
2026-09-27 02:35:27,431 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 02:35:27,432 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcH4HTG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41113/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 02:35:27,667 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 02:35:27,667 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcH4HTG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41113/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 02:35:27,917 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 02:35:27,918 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcH4HTG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41113/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 02:35:28,178 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 02:35:28,179 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcH4HTG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41113/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 02:35:28,419 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcH4HTG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41113/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 02:35:28,637 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcH4HTG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41113/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-grlk.c.gcp-upstream.internal';"
2026-09-27 02:35:28,854 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 02:35:28,855 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcH4HTG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41113/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 02:35:29,096 [I] DB_PATCHES have already been applied
I20260927 02:35:29.102077 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskcH4HTG/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskcH4HTG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 02:35:29.102147 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 02:35:29.105440 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:41749
Sep 27, 2026 2:35:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 2:35:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 2:35:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 2:35:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 2:35:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 2:35:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 2:35:29 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 02:35:30.115391 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.117s user 0.000s sys 0.008s
Sep 27, 2026 2:35:48 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 2:35:48 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 2:35:48 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 2:35:48 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 2:35:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 2:35:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 2:35:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 2:35:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 2:35:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 2:35:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 2:35:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 2:35:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 2:35:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 2:35:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 2:35:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 2:35:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 2:35:48 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 2:35:49 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 02:35:50.323275 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 02:35:50 dist-test-slave-grlk krb5kdc[12150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476550, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 02:35:50.402963 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 02:35:50.411357 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 02:35:51.128754 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 02:35:51.128819 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ranger-kms
I20260927 02:35:51.128836 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:41113
2026-09-27 02:35:51,277 [I] DB FLAVOR :POSTGRES
2026-09-27 02:35:51,277 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 02:35:51,278 [I] Checking connection
2026-09-27 02:35:51,278 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcH4HTG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41113/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 02:35:51,524 [I] connection success
2026-09-27 02:35:51,525 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 02:35:51,525 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 02:35:51,525 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcH4HTG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41113/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 02:35:51,777 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 02:35:51,777 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 02:35:51,777 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 02:35:51,777 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcH4HTG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41113/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 02:35:52,011 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 02:35:52.428661 392 mini_ranger_kms.cc:310] Created kms service
I20260927 02:35:52.637158 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 02:35:52.714066 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 02:35:52 dist-test-slave-grlk krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476550, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:35:53.098035 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 02:35:53.098269 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskcH4HTG/build/debug/bin/postgresql.jar:/tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskcH4HTG/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 02:35:53.098296 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 02:35:53.101958 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12543
I20260927 02:35:53.102013 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:38227
Sep 27, 2026 2:35:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 2:35:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 2:35:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 2:35:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 2:35:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 2:35:54 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 02:35:55.152168 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.749s user 0.011s sys 0.122s
I20260927 02:35:55.152302 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 02:35:55.152390 392 mini_ranger_kms.cc:401] 127.0.98.20:38227/kms/v1/keys
Sep 27 02:35:55 dist-test-slave-grlk krb5kdc[12150](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790476555, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 02:35:55 dist-test-slave-grlk krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476555, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 02:35:55 dist-test-slave-grlk krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476555, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 02:35:58 dist-test-slave-grlk krb5kdc[12150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:35:58.055389 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34871
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34871
--ranger_config_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38227/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:35:58.161031 12594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:35:58.161296 12594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:35:58.161348 12594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:35:58.165479 12594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:35:58.165555 12594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:35:58.165581 12594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:35:58.165601 12594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:35:58.165632 12594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:35:58.171288 12594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:38227/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34871
--ranger_config_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:35:58.172487 12594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:35:58.173563 12594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:35:58.182052 12602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:35:58.182236 12599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:35:58.182183 12600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:35:58.182287 12594 server_base.cc:1034] running on GCE node
I20260927 02:35:58.182965 12594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:35:58.184056 12594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:35:58.185197 12594 hybrid_clock.cc:648] HybridClock initialized: now 1790476558185182 us; error 36 us; skew 500 ppm
Sep 27 02:35:58 dist-test-slave-grlk krb5kdc[12150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:35:58.192025 12594 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:35:58.192816 12594 webserver.cc:460] Webserver started at http://127.0.98.62:42215/ using document root <none> and password file <none>
I20260927 02:35:58.193336 12594 fs_manager.cc:359] Metadata directory not provided
I20260927 02:35:58.193389 12594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:35:58.193575 12594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 02:35:58 dist-test-slave-grlk krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:35:58.241225 12594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "e35e5f8d6bec46c2a790209417aec057"
format_stamp: "Formatted at 2026-09-27 02:35:58 on dist-test-slave-grlk"
server_key: "bcff566336906df60455ffed378fe317"
server_key_iv: "1e2f1234cae503fe199cde4149727320"
server_key_version: "kuduclusterkey@0"
I20260927 02:35:58.241904 12594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "e35e5f8d6bec46c2a790209417aec057"
format_stamp: "Formatted at 2026-09-27 02:35:58 on dist-test-slave-grlk"
server_key: "bcff566336906df60455ffed378fe317"
server_key_iv: "1e2f1234cae503fe199cde4149727320"
server_key_version: "kuduclusterkey@0"
I20260927 02:35:58.245507 12594 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:35:58.264194 12609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:35:58.265969 12594 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260927 02:35:58.266192 12594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "e35e5f8d6bec46c2a790209417aec057"
format_stamp: "Formatted at 2026-09-27 02:35:58 on dist-test-slave-grlk"
server_key: "bcff566336906df60455ffed378fe317"
server_key_iv: "1e2f1234cae503fe199cde4149727320"
server_key_version: "kuduclusterkey@0"
I20260927 02:35:58.266343 12594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:35:58.302728 12594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:35:58.305469 12594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:35:58.305629 12594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:35:58.312664 12594 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34871
I20260927 02:35:58.312682 12661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34871 every 8 connection(s)
I20260927 02:35:58.313689 12594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:35:58.316298 12662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:35:58.321575 12662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e35e5f8d6bec46c2a790209417aec057: Bootstrap starting.
I20260927 02:35:58.323737 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 12594
I20260927 02:35:58.323866 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:35:58.323956 12662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e35e5f8d6bec46c2a790209417aec057: Neither blocks nor log segments found. Creating new log.
I20260927 02:35:58.324177 392 external_mini_cluster.cc:1428] Setting key 96d57c491cba47dc2e7fd5c71da5c93d
I20260927 02:35:58.324693 12662 log.cc:824] T 00000000000000000000000000000000 P e35e5f8d6bec46c2a790209417aec057: Log is configured to *not* fsync() on all Append() calls
I20260927 02:35:58.326988 12662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e35e5f8d6bec46c2a790209417aec057: No bootstrap required, opened a new log
I20260927 02:35:58.330802 12662 raft_consensus.cc:348] T 00000000000000000000000000000000 P e35e5f8d6bec46c2a790209417aec057 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e35e5f8d6bec46c2a790209417aec057" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34871 } }
I20260927 02:35:58.331019 12662 raft_consensus.cc:374] T 00000000000000000000000000000000 P e35e5f8d6bec46c2a790209417aec057 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:35:58.331094 12662 raft_consensus.cc:729] T 00000000000000000000000000000000 P e35e5f8d6bec46c2a790209417aec057 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e35e5f8d6bec46c2a790209417aec057, State: Initialized, Role: FOLLOWER
I20260927 02:35:58.331609 12662 consensus_queue.cc:260] T 00000000000000000000000000000000 P e35e5f8d6bec46c2a790209417aec057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e35e5f8d6bec46c2a790209417aec057" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34871 } }
I20260927 02:35:58.331765 12662 raft_consensus.cc:388] T 00000000000000000000000000000000 P e35e5f8d6bec46c2a790209417aec057 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:35:58.331851 12662 raft_consensus.cc:482] T 00000000000000000000000000000000 P e35e5f8d6bec46c2a790209417aec057 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:35:58.331947 12662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e35e5f8d6bec46c2a790209417aec057 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:35:58.332851 12662 raft_consensus.cc:504] T 00000000000000000000000000000000 P e35e5f8d6bec46c2a790209417aec057 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e35e5f8d6bec46c2a790209417aec057" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34871 } }
I20260927 02:35:58.333199 12662 leader_election.cc:302] T 00000000000000000000000000000000 P e35e5f8d6bec46c2a790209417aec057 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e35e5f8d6bec46c2a790209417aec057; no voters:
I20260927 02:35:58.333593 12662 leader_election.cc:288] T 00000000000000000000000000000000 P e35e5f8d6bec46c2a790209417aec057 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:35:58.333671 12667 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e35e5f8d6bec46c2a790209417aec057 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:35:58.334945 12662 sys_catalog.cc:564] T 00000000000000000000000000000000 P e35e5f8d6bec46c2a790209417aec057 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:35:58.334993 12667 raft_consensus.cc:686] T 00000000000000000000000000000000 P e35e5f8d6bec46c2a790209417aec057 [term 1 LEADER]: Becoming Leader. State: Replica: e35e5f8d6bec46c2a790209417aec057, State: Running, Role: LEADER
I20260927 02:35:58.335299 12667 consensus_queue.cc:237] T 00000000000000000000000000000000 P e35e5f8d6bec46c2a790209417aec057 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e35e5f8d6bec46c2a790209417aec057" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34871 } }
Sep 27 02:35:58 dist-test-slave-grlk krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:35:58.336413 12667 sys_catalog.cc:455] T 00000000000000000000000000000000 P e35e5f8d6bec46c2a790209417aec057 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e35e5f8d6bec46c2a790209417aec057. Latest consensus state: current_term: 1 leader_uuid: "e35e5f8d6bec46c2a790209417aec057" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e35e5f8d6bec46c2a790209417aec057" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34871 } } }
I20260927 02:35:58.336515 12667 sys_catalog.cc:458] T 00000000000000000000000000000000 P e35e5f8d6bec46c2a790209417aec057 [sys.catalog]: This master's current role is: LEADER
I20260927 02:35:58.336838 12668 sys_catalog.cc:455] T 00000000000000000000000000000000 P e35e5f8d6bec46c2a790209417aec057 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e35e5f8d6bec46c2a790209417aec057" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e35e5f8d6bec46c2a790209417aec057" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34871 } } }
I20260927 02:35:58.336966 12668 sys_catalog.cc:458] T 00000000000000000000000000000000 P e35e5f8d6bec46c2a790209417aec057 [sys.catalog]: This master's current role is: LEADER
I20260927 02:35:58.337093 12679 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:35:58.340093 12679 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:35:58.337093 12662 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 02:35:58.342288 12666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:35:58.325395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60988 (local address 127.0.98.62:34871)
0927 02:35:58.325839 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0927 02:35:58.325857 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 02:35:58.327152 (+ 1295us) server_negotiation.cc:407] Connection header received
0927 02:35:58.328012 (+ 860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:35:58.328044 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:35:58.328435 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:35:58.328794 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:35:58.329659 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:35:58.331171 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:35:58.332331 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:35:58.333045 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:35:58.333184 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:35:58.336100 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:35:58.336106 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:35:58.336122 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:35:58.336153 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:35:58.338700 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:35:58.339189 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:35:58.339195 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:35:58.339201 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:35:58.339325 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:35:58.339605 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:35:58.339609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:35:58.339610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:35:58.340097 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:35:58.340266 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 02:35:58.341100 (+ 834us) server_negotiation.cc:299] Negotiation successful
0927 02:35:58.341366 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":128,"threads_started":1}
I20260927 02:35:58.350596 12684 mvcc.cc:205] Tried to move back new op lower bound from 7333791982998269952 to 7333791982942117888. Current Snapshot: MvccSnapshot[applied={T|T < 7333791982998269952}]
I20260927 02:35:58.352819 12679 catalog_manager.cc:1347] Generated new cluster ID: 78ead5dc2a75449dab93f468c4bb82d3
I20260927 02:35:58.352877 12679 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:35:58.397569 12679 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:35:58.399001 12679 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:35:58.404170 12679 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e35e5f8d6bec46c2a790209417aec057: Generated new TSK 0
I20260927 02:35:58.405023 12679 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:35:59.067948 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38227/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34871
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:35:59.211395 12707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:35:59.211688 12707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:35:59.211748 12707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:35:59.216785 12707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:35:59.216915 12707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:35:59.217049 12707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:35:59.224047 12707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:38227/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:35:59.225153 12707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:35:59.226141 12707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:35:59.237572 12714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:35:59.237782 12707 server_base.cc:1034] running on GCE node
W20260927 02:35:59.237830 12715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:35:59.238003 12717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:35:59.238656 12707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:35:59.239427 12707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:35:59.240625 12707 hybrid_clock.cc:648] HybridClock initialized: now 1790476559240606 us; error 35 us; skew 500 ppm
Sep 27 02:35:59 dist-test-slave-grlk krb5kdc[12150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:35:59.243784 12707 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:35:59.244959 12707 webserver.cc:460] Webserver started at http://127.0.98.1:37471/ using document root <none> and password file <none>
I20260927 02:35:59.245678 12707 fs_manager.cc:359] Metadata directory not provided
I20260927 02:35:59.245770 12707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:35:59.246059 12707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 02:35:59 dist-test-slave-grlk krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:35:59.262394 12707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-0/data/instance:
uuid: "5d2724da61d341379c6d243f088d3ed5"
format_stamp: "Formatted at 2026-09-27 02:35:59 on dist-test-slave-grlk"
server_key: "9ba33bfa519eef9e40876b53db3c8d11"
server_key_iv: "0f48f40c3556c8d77536c874e5a2eb25"
server_key_version: "kuduclusterkey@0"
I20260927 02:35:59.263119 12707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5d2724da61d341379c6d243f088d3ed5"
format_stamp: "Formatted at 2026-09-27 02:35:59 on dist-test-slave-grlk"
server_key: "9ba33bfa519eef9e40876b53db3c8d11"
server_key_iv: "0f48f40c3556c8d77536c874e5a2eb25"
server_key_version: "kuduclusterkey@0"
I20260927 02:35:59.267973 12707 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 02:35:59.302093 12724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:35:59.303834 12707 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.002s sys 0.001s
I20260927 02:35:59.303979 12707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-0/wal
uuid: "5d2724da61d341379c6d243f088d3ed5"
format_stamp: "Formatted at 2026-09-27 02:35:59 on dist-test-slave-grlk"
server_key: "9ba33bfa519eef9e40876b53db3c8d11"
server_key_iv: "0f48f40c3556c8d77536c874e5a2eb25"
server_key_version: "kuduclusterkey@0"
I20260927 02:35:59.304132 12707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:35:59.336707 12707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:35:59.340250 12707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:35:59.340451 12707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:35:59.341212 12707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:35:59.342234 12707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:35:59.342295 12707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:35:59.342348 12707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:35:59.342420 12707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260927 02:35:59.353515 12707 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44329
I20260927 02:35:59.354730 12707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:35:59.355494 12839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44329 every 8 connection(s)
I20260927 02:35:59.359097 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 12707
I20260927 02:35:59.359200 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:35:59.359474 392 external_mini_cluster.cc:1428] Setting key b18911d07bb4c5b46aad4179f116a73b
Sep 27 02:35:59 dist-test-slave-grlk krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 02:35:59.375629 12842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:35:59.357187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44269 (local address 127.0.98.62:34871)
0927 02:35:59.357470 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0927 02:35:59.357478 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:35:59.357951 (+ 473us) server_negotiation.cc:407] Connection header received
0927 02:35:59.359172 (+ 1221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:35:59.359176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:35:59.359232 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:35:59.359289 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:35:59.360831 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:35:59.361736 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:35:59.363389 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:35:59.363754 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:35:59.363823 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:35:59.370886 (+ 7063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:35:59.370890 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:35:59.370893 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:35:59.370919 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:35:59.372790 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:35:59.373591 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:35:59.373595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:35:59.373597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:35:59.373696 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:35:59.374221 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:35:59.374225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:35:59.374226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:35:59.374464 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:35:59.374546 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 02:35:59.375318 (+ 772us) server_negotiation.cc:299] Negotiation successful
0927 02:35:59.375450 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":101,"threads_started":1}
I20260927 02:35:59.376667 12843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:35:59.357272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34871 (local address 127.0.98.1:44269)
0927 02:35:59.357800 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0927 02:35:59.357854 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 02:35:59.358919 (+ 1065us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:35:59.359691 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:35:59.359701 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:35:59.360049 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:35:59.360667 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:35:59.360684 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:35:59.361885 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:35:59.361888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:35:59.363260 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:35:59.363270 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:35:59.364019 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:35:59.364023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:35:59.364204 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:35:59.369045 (+ 4841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:35:59.369066 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:35:59.370677 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:35:59.373110 (+ 2433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:35:59.373117 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:35:59.373135 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:35:59.373477 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:35:59.373812 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:35:59.373815 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:35:59.373817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:35:59.374102 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:35:59.374627 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:35:59.374634 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:35:59.375004 (+ 370us) client_negotiation.cc:769] Sending connection context
0927 02:35:59.375254 (+ 250us) client_negotiation.cc:240] Negotiation successful
0927 02:35:59.375601 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":159,"threads_started":1}
I20260927 02:35:59.378368 12840 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34871
I20260927 02:35:59.378626 12840 heartbeater.cc:464] Registering TS with master...
I20260927 02:35:59.379221 12840 heartbeater.cc:511] Master 127.0.98.62:34871 requested a full tablet report, sending...
I20260927 02:35:59.381011 12626 ts_manager.cc:194] Registered new tserver with Master: 5d2724da61d341379c6d243f088d3ed5 (127.0.98.1:44329)
I20260927 02:35:59.382567 12626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44269
Sep 27 02:35:59 dist-test-slave-grlk krb5kdc[12150](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790476559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:35:59.392400 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38227/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34871
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:35:59.518554 12846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:35:59.518808 12846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:35:59.518865 12846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:35:59.524250 12846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:35:59.524319 12846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:35:59.524415 12846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:35:59.529984 12846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:38227/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:35:59.531222 12846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:35:59.532222 12846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:35:59.542374 12855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:35:59.542923 12853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:35:59.543169 12852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:35:59.543360 12846 server_base.cc:1034] running on GCE node
I20260927 02:35:59.543875 12846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:35:59.544610 12846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:35:59.547894 12846 hybrid_clock.cc:648] HybridClock initialized: now 1790476559547821 us; error 80 us; skew 500 ppm
Sep 27 02:35:59 dist-test-slave-grlk krb5kdc[12150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:35:59.551483 12846 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:35:59.552539 12846 webserver.cc:460] Webserver started at http://127.0.98.2:37105/ using document root <none> and password file <none>
I20260927 02:35:59.553222 12846 fs_manager.cc:359] Metadata directory not provided
I20260927 02:35:59.553290 12846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:35:59.553519 12846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 02:35:59 dist-test-slave-grlk krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:35:59.569093 12846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-1/data/instance:
uuid: "c269b7cd26b546dd8f92c51d5cfc98f6"
format_stamp: "Formatted at 2026-09-27 02:35:59 on dist-test-slave-grlk"
server_key: "69413983cf979996a591856fb840ae3c"
server_key_iv: "3af38f2420e9edce12608319b2c9a646"
server_key_version: "kuduclusterkey@0"
I20260927 02:35:59.569995 12846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c269b7cd26b546dd8f92c51d5cfc98f6"
format_stamp: "Formatted at 2026-09-27 02:35:59 on dist-test-slave-grlk"
server_key: "69413983cf979996a591856fb840ae3c"
server_key_iv: "3af38f2420e9edce12608319b2c9a646"
server_key_version: "kuduclusterkey@0"
I20260927 02:35:59.574276 12846 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 02:35:59.595929 12862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:35:59.597651 12846 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260927 02:35:59.597841 12846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-1/wal
uuid: "c269b7cd26b546dd8f92c51d5cfc98f6"
format_stamp: "Formatted at 2026-09-27 02:35:59 on dist-test-slave-grlk"
server_key: "69413983cf979996a591856fb840ae3c"
server_key_iv: "3af38f2420e9edce12608319b2c9a646"
server_key_version: "kuduclusterkey@0"
I20260927 02:35:59.598057 12846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:35:59.629926 12846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:35:59.633493 12846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:35:59.633708 12846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:35:59.634461 12846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:35:59.635471 12846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:35:59.635568 12846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:35:59.635663 12846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:35:59.635721 12846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:35:59.645311 12846 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42501
I20260927 02:35:59.645470 12975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42501 every 8 connection(s)
I20260927 02:35:59.646376 12846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:35:59.651484 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 12846
I20260927 02:35:59.651618 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:35:59.651922 392 external_mini_cluster.cc:1428] Setting key 436b13a9e5bdb3bc8fbbaf45926a8416
Sep 27 02:35:59 dist-test-slave-grlk krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:35:59.662415 12978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:35:59.648717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34871 (local address 127.0.98.2:47345)
0927 02:35:59.649831 (+ 1114us) negotiation.cc:107] Waiting for socket to connect
0927 02:35:59.649885 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 02:35:59.650863 (+ 978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:35:59.651451 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:35:59.651460 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:35:59.651775 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:35:59.652396 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:35:59.652416 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:35:59.653446 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:35:59.653449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:35:59.654706 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:35:59.654717 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:35:59.655354 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:35:59.655356 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:35:59.655493 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:35:59.656050 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:35:59.656065 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:35:59.657456 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:35:59.659320 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:35:59.659326 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:35:59.659339 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:35:59.659664 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:35:59.659952 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:35:59.659955 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:35:59.659957 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:35:59.660118 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:35:59.660596 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:35:59.660602 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:35:59.660907 (+ 305us) client_negotiation.cc:769] Sending connection context
0927 02:35:59.661141 (+ 234us) client_negotiation.cc:240] Negotiation successful
0927 02:35:59.661449 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":916,"thread_start_us":162,"threads_started":1}
I20260927 02:35:59.663584 12842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:35:59.648255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47345 (local address 127.0.98.62:34871)
0927 02:35:59.648396 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 02:35:59.648401 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:35:59.650001 (+ 1600us) server_negotiation.cc:407] Connection header received
0927 02:35:59.651097 (+ 1096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:35:59.651101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:35:59.651153 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:35:59.651258 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:35:59.652556 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:35:59.653292 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:35:59.654858 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:35:59.655148 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:35:59.655206 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:35:59.657607 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:35:59.657611 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:35:59.657614 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:35:59.657642 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:35:59.659196 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:35:59.659772 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:35:59.659775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:35:59.659777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:35:59.659859 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:35:59.660228 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:35:59.660231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:35:59.660233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:35:59.660453 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:35:59.660543 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 02:35:59.663305 (+ 2762us) server_negotiation.cc:299] Negotiation successful
0927 02:35:59.663425 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:35:59.664440 12976 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34871
I20260927 02:35:59.664673 12976 heartbeater.cc:464] Registering TS with master...
I20260927 02:35:59.665207 12976 heartbeater.cc:511] Master 127.0.98.62:34871 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 02:35:59.666213 12626 ts_manager.cc:194] Registered new tserver with Master: c269b7cd26b546dd8f92c51d5cfc98f6 (127.0.98.2:42501)
I20260927 02:35:59.666823 12626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47345
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:35:59.688727 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38227/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34871
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 02:35:59 dist-test-slave-grlk krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260927 02:35:59.847065 12981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:35:59.847482 12981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:35:59.847584 12981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:35:59.853952 12981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:35:59.854246 12981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:35:59.854422 12981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:35:59.863052 12981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:38227/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:35:59.864737 12981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:35:59.866130 12981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:35:59.883550 12981 server_base.cc:1034] running on GCE node
W20260927 02:35:59.883908 12989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:35:59.889070 12986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:35:59.889081 12987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:35:59.889658 12981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:35:59.890424 12981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:35:59.897769 12981 hybrid_clock.cc:648] HybridClock initialized: now 1790476559897693 us; error 90 us; skew 500 ppm
Sep 27 02:35:59 dist-test-slave-grlk krb5kdc[12150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:35:59.901239 12981 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:35:59.902248 12981 webserver.cc:460] Webserver started at http://127.0.98.3:38607/ using document root <none> and password file <none>
I20260927 02:35:59.902878 12981 fs_manager.cc:359] Metadata directory not provided
I20260927 02:35:59.902940 12981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:35:59.903160 12981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 02:35:59 dist-test-slave-grlk krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:35:59.918670 12981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-2/data/instance:
uuid: "930b2ebba1d444f19c396718c61d45c9"
format_stamp: "Formatted at 2026-09-27 02:35:59 on dist-test-slave-grlk"
server_key: "21059ae984ef48c25752a4678569ded8"
server_key_iv: "d6ad97408df9fff62e9de5991440d9b6"
server_key_version: "kuduclusterkey@0"
I20260927 02:35:59.919569 12981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-2/wal/instance:
uuid: "930b2ebba1d444f19c396718c61d45c9"
format_stamp: "Formatted at 2026-09-27 02:35:59 on dist-test-slave-grlk"
server_key: "21059ae984ef48c25752a4678569ded8"
server_key_iv: "d6ad97408df9fff62e9de5991440d9b6"
server_key_version: "kuduclusterkey@0"
I20260927 02:35:59.926164 12981 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260927 02:35:59.941685 12996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:35:59.943500 12981 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:35:59.943717 12981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-2/wal
uuid: "930b2ebba1d444f19c396718c61d45c9"
format_stamp: "Formatted at 2026-09-27 02:35:59 on dist-test-slave-grlk"
server_key: "21059ae984ef48c25752a4678569ded8"
server_key_iv: "d6ad97408df9fff62e9de5991440d9b6"
server_key_version: "kuduclusterkey@0"
I20260927 02:35:59.943959 12981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:35:59.973596 12981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:35:59.976949 12981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:35:59.977131 12981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:35:59.977816 12981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:35:59.978826 12981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:35:59.978900 12981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:35:59.978953 12981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:35:59.978982 12981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:35:59.989516 12981 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38845
I20260927 02:35:59.989567 13109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38845 every 8 connection(s)
I20260927 02:35:59.990694 12981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:35:59.991670 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 12981
I20260927 02:35:59.991779 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:35:59.992085 392 external_mini_cluster.cc:1428] Setting key 0b2fb0c3aec562e87d788e4daf43f4f2
Sep 27 02:35:59 dist-test-slave-grlk krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:36:00.006716 12842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:35:59.993102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36383 (local address 127.0.98.62:34871)
0927 02:35:59.993226 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 02:35:59.993231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:35:59.994143 (+ 912us) server_negotiation.cc:407] Connection header received
0927 02:35:59.995342 (+ 1199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:35:59.995346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:35:59.995396 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:35:59.995492 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:35:59.996809 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:35:59.997647 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:35:59.999277 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:35:59.999604 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:35:59.999700 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:00.002554 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:36:00.002557 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:36:00.002560 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:36:00.002587 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:36:00.004309 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:00.004891 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:00.004894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:00.004896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:00.005027 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:00.005431 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:00.005435 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:00.005436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:00.005652 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:36:00.005755 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:00.006425 (+ 670us) server_negotiation.cc:299] Negotiation successful
0927 02:36:00.006552 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 02:36:00.007644 13112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:35:59.993440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34871 (local address 127.0.98.3:36383)
0927 02:35:59.993984 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0927 02:35:59.994041 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 02:35:59.995123 (+ 1082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:35:59.995655 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:35:59.995665 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:35:59.996017 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:35:59.996650 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:35:59.996674 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:35:59.997770 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:35:59.997774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:35:59.999156 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:35:59.999167 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:35:59.999855 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:35:59.999859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:00.000042 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:00.000691 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:36:00.000710 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:36:00.002372 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:36:00.004440 (+ 2068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:00.004446 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:00.004460 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:00.004781 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:00.005136 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:00.005140 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:00.005142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:00.005321 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:00.005807 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:36:00.005814 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:36:00.006146 (+ 332us) client_negotiation.cc:769] Sending connection context
0927 02:36:00.006365 (+ 219us) client_negotiation.cc:240] Negotiation successful
0927 02:36:00.006677 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":137,"threads_started":1}
I20260927 02:36:00.009086 13110 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34871
I20260927 02:36:00.009369 13110 heartbeater.cc:464] Registering TS with master...
I20260927 02:36:00.010052 13110 heartbeater.cc:511] Master 127.0.98.62:34871 requested a full tablet report, sending...
I20260927 02:36:00.011161 12626 ts_manager.cc:194] Registered new tserver with Master: 930b2ebba1d444f19c396718c61d45c9 (127.0.98.3:38845)
I20260927 02:36:00.011906 12626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36383
I20260927 02:36:00.017509 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:36:00.018373 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 12707
I20260927 02:36:00.025779 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 12846
I20260927 02:36:00.030182 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 12981
I20260927 02:36:00.034684 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 12594
I20260927 02:36:00.039969 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34871
--webserver_interface=127.0.98.62
--webserver_port=42215
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34871
--ranger_config_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38227/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:38227/kms with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:36:00.159435 13113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:36:00.159787 13113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:36:00.159845 13113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:36:00.165623 13113 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:36:00.165695 13113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:36:00.165714 13113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:36:00.165729 13113 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:36:00.165750 13113 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:36:00.171268 13113 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:38227/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34871
--ranger_config_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:36:00.172287 13113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:36:00.173264 13113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:36:00.181260 13119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:36:00.181272 13118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:36:00.181406 13113 server_base.cc:1034] running on GCE node
W20260927 02:36:00.181254 13121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:36:00.182041 13113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:36:00.183076 13113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:36:00.184242 13113 hybrid_clock.cc:648] HybridClock initialized: now 1790476560184219 us; error 31 us; skew 500 ppm
Sep 27 02:36:00 dist-test-slave-grlk krb5kdc[12150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:36:00.186746 13113 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:36:00.187556 13113 webserver.cc:460] Webserver started at http://127.0.98.62:42215/ using document root <none> and password file <none>
I20260927 02:36:00.188074 13113 fs_manager.cc:359] Metadata directory not provided
I20260927 02:36:00.188143 13113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 02:36:00 dist-test-slave-grlk krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:36:00.248582 13113 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.002s
I20260927 02:36:00.251266 13129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:00.252790 13113 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 02:36:00.252957 13113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "e35e5f8d6bec46c2a790209417aec057"
format_stamp: "Formatted at 2026-09-27 02:35:58 on dist-test-slave-grlk"
server_key: "bcff566336906df60455ffed378fe317"
server_key_iv: "1e2f1234cae503fe199cde4149727320"
server_key_version: "kuduclusterkey@0"
I20260927 02:36:00.253664 13113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:36:00.284375 13113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:36:00.286967 13113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:36:00.287110 13113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:36:00.293684 13113 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34871
I20260927 02:36:00.293707 13181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34871 every 8 connection(s)
I20260927 02:36:00.294684 13113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:36:00.298875 13182 sys_catalog.cc:263] Verifying existing consensus state
I20260927 02:36:00.299716 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 13113
I20260927 02:36:00.300195 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38227/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44329
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34871
--webserver_port=37471
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:38227/kms with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:36:00.300792 13182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e35e5f8d6bec46c2a790209417aec057: Bootstrap starting.
I20260927 02:36:00.316582 13182 log.cc:824] T 00000000000000000000000000000000 P e35e5f8d6bec46c2a790209417aec057: Log is configured to *not* fsync() on all Append() calls
I20260927 02:36:00.322767 13182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e35e5f8d6bec46c2a790209417aec057: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:36:00.323117 13182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e35e5f8d6bec46c2a790209417aec057: Bootstrap complete.
I20260927 02:36:00.326996 13182 raft_consensus.cc:348] T 00000000000000000000000000000000 P e35e5f8d6bec46c2a790209417aec057 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e35e5f8d6bec46c2a790209417aec057" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34871 } }
I20260927 02:36:00.327404 13182 raft_consensus.cc:729] T 00000000000000000000000000000000 P e35e5f8d6bec46c2a790209417aec057 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e35e5f8d6bec46c2a790209417aec057, State: Initialized, Role: FOLLOWER
I20260927 02:36:00.327875 13182 consensus_queue.cc:260] T 00000000000000000000000000000000 P e35e5f8d6bec46c2a790209417aec057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e35e5f8d6bec46c2a790209417aec057" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34871 } }
I20260927 02:36:00.327987 13182 raft_consensus.cc:388] T 00000000000000000000000000000000 P e35e5f8d6bec46c2a790209417aec057 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:36:00.328043 13182 raft_consensus.cc:482] T 00000000000000000000000000000000 P e35e5f8d6bec46c2a790209417aec057 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:36:00.328128 13182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e35e5f8d6bec46c2a790209417aec057 [term 1 FOLLOWER]: Advancing to term 2
I20260927 02:36:00.329873 13182 raft_consensus.cc:504] T 00000000000000000000000000000000 P e35e5f8d6bec46c2a790209417aec057 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e35e5f8d6bec46c2a790209417aec057" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34871 } }
I20260927 02:36:00.330160 13182 leader_election.cc:302] T 00000000000000000000000000000000 P e35e5f8d6bec46c2a790209417aec057 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e35e5f8d6bec46c2a790209417aec057; no voters:
I20260927 02:36:00.330437 13182 leader_election.cc:288] T 00000000000000000000000000000000 P e35e5f8d6bec46c2a790209417aec057 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 02:36:00.330602 13186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e35e5f8d6bec46c2a790209417aec057 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 02:36:00.330892 13186 raft_consensus.cc:686] T 00000000000000000000000000000000 P e35e5f8d6bec46c2a790209417aec057 [term 2 LEADER]: Becoming Leader. State: Replica: e35e5f8d6bec46c2a790209417aec057, State: Running, Role: LEADER
I20260927 02:36:00.331245 13186 consensus_queue.cc:237] T 00000000000000000000000000000000 P e35e5f8d6bec46c2a790209417aec057 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e35e5f8d6bec46c2a790209417aec057" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34871 } }
I20260927 02:36:00.331490 13182 sys_catalog.cc:564] T 00000000000000000000000000000000 P e35e5f8d6bec46c2a790209417aec057 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:36:00.332494 13188 sys_catalog.cc:455] T 00000000000000000000000000000000 P e35e5f8d6bec46c2a790209417aec057 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e35e5f8d6bec46c2a790209417aec057. Latest consensus state: current_term: 2 leader_uuid: "e35e5f8d6bec46c2a790209417aec057" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e35e5f8d6bec46c2a790209417aec057" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34871 } } }
I20260927 02:36:00.332607 13188 sys_catalog.cc:458] T 00000000000000000000000000000000 P e35e5f8d6bec46c2a790209417aec057 [sys.catalog]: This master's current role is: LEADER
I20260927 02:36:00.333252 13187 sys_catalog.cc:455] T 00000000000000000000000000000000 P e35e5f8d6bec46c2a790209417aec057 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e35e5f8d6bec46c2a790209417aec057" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e35e5f8d6bec46c2a790209417aec057" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34871 } } }
I20260927 02:36:00.333334 13187 sys_catalog.cc:458] T 00000000000000000000000000000000 P e35e5f8d6bec46c2a790209417aec057 [sys.catalog]: This master's current role is: LEADER
I20260927 02:36:00.333539 13182 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 02:36:00.333621 13196 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:36:00.339461 13196 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:36:00.340284 13196 catalog_manager.cc:1259] Loaded cluster ID: 78ead5dc2a75449dab93f468c4bb82d3
I20260927 02:36:00.340377 13196 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:36:00.343746 13196 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:36:00.344521 13196 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e35e5f8d6bec46c2a790209417aec057: Loaded TSK: 0
I20260927 02:36:00.345793 13196 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 02:36:00.425820 13184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:36:00.426141 13184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:36:00.426205 13184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:36:00.432320 13184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:36:00.432447 13184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:36:00.432564 13184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:36:00.440267 13184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:38227/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44329
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37471
--tserver_master_addrs=127.0.98.62:34871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:36:00.441653 13184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:36:00.442852 13184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:36:00.454697 13221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:36:00.454764 13223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:36:00.455082 13184 server_base.cc:1034] running on GCE node
W20260927 02:36:00.455014 13220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:36:00.455636 13184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:36:00.456400 13184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:36:00.457592 13184 hybrid_clock.cc:648] HybridClock initialized: now 1790476560457564 us; error 41 us; skew 500 ppm
Sep 27 02:36:00 dist-test-slave-grlk krb5kdc[12150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:36:00.461102 13184 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:36:00.462174 13184 webserver.cc:460] Webserver started at http://127.0.98.1:37471/ using document root <none> and password file <none>
I20260927 02:36:00.462893 13184 fs_manager.cc:359] Metadata directory not provided
I20260927 02:36:00.463027 13184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 02:36:00 dist-test-slave-grlk krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:36:00.486588 13184 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 02:36:00.489734 13231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:00.494093 13184 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.001s
I20260927 02:36:00.494227 13184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-0/wal
uuid: "5d2724da61d341379c6d243f088d3ed5"
format_stamp: "Formatted at 2026-09-27 02:35:59 on dist-test-slave-grlk"
server_key: "9ba33bfa519eef9e40876b53db3c8d11"
server_key_iv: "0f48f40c3556c8d77536c874e5a2eb25"
server_key_version: "kuduclusterkey@0"
I20260927 02:36:00.494920 13184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:36:00.529872 13184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:36:00.533790 13184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:36:00.533995 13184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:36:00.534739 13184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:36:00.535815 13184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:36:00.535877 13184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:00.535945 13184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:36:00.536007 13184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260927 02:36:00.549594 13184 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44329
I20260927 02:36:00.549685 13344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44329 every 8 connection(s)
I20260927 02:36:00.550971 13184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:36:00.559525 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 13184
I20260927 02:36:00.560019 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38227/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42501
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34871
--webserver_port=37105
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:38227/kms with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 02:36:00 dist-test-slave-grlk krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:36:00.581748 13348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:00.558030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34871 (local address 127.0.98.1:54261)
0927 02:36:00.558549 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:00.558609 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:00.559594 (+ 985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:00.560802 (+ 1208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:36:00.560810 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:36:00.561155 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:36:00.561769 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:00.561788 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:00.563667 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:00.563671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:00.565052 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:00.565063 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:00.566049 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:00.566054 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:00.566243 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:00.569074 (+ 2831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:36:00.569094 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:36:00.570942 (+ 1848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:36:00.573487 (+ 2545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:00.573493 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:00.573506 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:00.574211 (+ 705us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:00.578901 (+ 4690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:00.578904 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:00.578906 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:00.579046 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:00.579812 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:36:00.579818 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:36:00.580133 (+ 315us) client_negotiation.cc:769] Sending connection context
0927 02:36:00.580328 (+ 195us) client_negotiation.cc:240] Negotiation successful
0927 02:36:00.580654 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":133,"threads_started":1}
I20260927 02:36:00.583534 13347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:00.557725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54261 (local address 127.0.98.62:34871)
0927 02:36:00.558140 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:00.558619 (+ 479us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:00.558735 (+ 116us) server_negotiation.cc:407] Connection header received
0927 02:36:00.560032 (+ 1297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:00.560042 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:00.560416 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:00.560708 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:36:00.561960 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:00.563515 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:00.565167 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:00.565847 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:00.566019 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:00.571159 (+ 5140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:36:00.571165 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:36:00.571180 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:36:00.571212 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:36:00.573362 (+ 2150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:00.574347 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:00.574353 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:00.574359 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:00.578793 (+ 4434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:00.579144 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:00.579146 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:00.579147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:00.579631 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:36:00.579758 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:00.582328 (+ 2570us) server_negotiation.cc:299] Negotiation successful
0927 02:36:00.582635 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":113,"threads_started":1}
I20260927 02:36:00.586647 13345 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34871
I20260927 02:36:00.586871 13345 heartbeater.cc:464] Registering TS with master...
I20260927 02:36:00.587417 13345 heartbeater.cc:511] Master 127.0.98.62:34871 requested a full tablet report, sending...
W20260927 02:36:00.589466 13345 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34871 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 02:36:00.589572 13345 heartbeater.cc:464] Registering TS with master...
I20260927 02:36:00.589715 13345 heartbeater.cc:511] Master 127.0.98.62:34871 requested a full tablet report, sending...
I20260927 02:36:00.590426 13345 heartbeater.cc:464] Registering TS with master...
I20260927 02:36:00.590583 13345 heartbeater.cc:511] Master 127.0.98.62:34871 requested a full tablet report, sending...
W20260927 02:36:00.591204 13345 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 02:36:00.700843 13349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:36:00.701136 13349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:36:00.701192 13349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:36:00.705324 13349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:36:00.705387 13349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:36:00.705451 13349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:36:00.710788 13349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:38227/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42501
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37105
--tserver_master_addrs=127.0.98.62:34871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:36:00.711853 13349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:36:00.712864 13349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:36:00.723213 13355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:36:00.723227 13354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:36:00.723516 13357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:36:00.723414 13349 server_base.cc:1034] running on GCE node
I20260927 02:36:00.723927 13349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:36:00.724524 13349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:36:00.725672 13349 hybrid_clock.cc:648] HybridClock initialized: now 1790476560725638 us; error 50 us; skew 500 ppm
Sep 27 02:36:00 dist-test-slave-grlk krb5kdc[12150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:36:00.729501 13349 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:36:00.730592 13349 webserver.cc:460] Webserver started at http://127.0.98.2:37105/ using document root <none> and password file <none>
I20260927 02:36:00.731262 13349 fs_manager.cc:359] Metadata directory not provided
I20260927 02:36:00.731359 13349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 02:36:00 dist-test-slave-grlk krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:36:00.753615 13349 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.003s sys 0.000s
I20260927 02:36:00.756778 13365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:00.758580 13349 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 02:36:00.758760 13349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-1/wal
uuid: "c269b7cd26b546dd8f92c51d5cfc98f6"
format_stamp: "Formatted at 2026-09-27 02:35:59 on dist-test-slave-grlk"
server_key: "69413983cf979996a591856fb840ae3c"
server_key_iv: "3af38f2420e9edce12608319b2c9a646"
server_key_version: "kuduclusterkey@0"
I20260927 02:36:00.759567 13349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:36:00.804702 13349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:36:00.808293 13349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:36:00.808568 13349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:36:00.809367 13349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:36:00.810451 13349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:36:00.810555 13349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:00.810657 13349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:36:00.810719 13349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:00.820986 13349 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42501
I20260927 02:36:00.821048 13479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42501 every 8 connection(s)
I20260927 02:36:00.822320 13349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:36:00.832158 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 13349
I20260927 02:36:00.832649 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38227/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:38845
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34871
--webserver_port=38607
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:38227/kms with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 02:36:00 dist-test-slave-grlk krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:36:00.839501 13347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:00.824504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55151 (local address 127.0.98.62:34871)
0927 02:36:00.824656 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:00.824660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:00.825494 (+ 834us) server_negotiation.cc:407] Connection header received
0927 02:36:00.826734 (+ 1240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:00.826737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:00.826800 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:00.826891 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:36:00.828280 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:00.829042 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:00.830799 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:00.831192 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:00.831303 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:00.835286 (+ 3983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:36:00.835290 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:36:00.835292 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:36:00.835321 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:36:00.837126 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:00.837724 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:00.837728 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:00.837730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:00.837817 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:00.838219 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:00.838223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:00.838225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:00.838440 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:36:00.838513 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:00.839213 (+ 700us) server_negotiation.cc:299] Negotiation successful
0927 02:36:00.839339 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 02:36:00.841046 13482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:00.824803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34871 (local address 127.0.98.2:55151)
0927 02:36:00.825320 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:00.825374 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:00.826484 (+ 1110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:00.827082 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:36:00.827091 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:36:00.827428 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:36:00.828093 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:00.828115 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:00.829207 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:00.829210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:00.830578 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:00.830590 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:00.831575 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:00.831579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:00.831767 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:00.832459 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:36:00.832479 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:36:00.835083 (+ 2604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:36:00.837259 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:00.837265 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:00.837281 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:00.837602 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:00.837931 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:00.837934 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:00.837936 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:00.838112 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:00.838591 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:36:00.838598 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:36:00.838927 (+ 329us) client_negotiation.cc:769] Sending connection context
0927 02:36:00.839715 (+ 788us) client_negotiation.cc:240] Negotiation successful
0927 02:36:00.840042 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":148,"threads_started":1}
I20260927 02:36:00.842279 13480 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34871
I20260927 02:36:00.842509 13480 heartbeater.cc:464] Registering TS with master...
I20260927 02:36:00.843060 13480 heartbeater.cc:511] Master 127.0.98.62:34871 requested a full tablet report, sending...
W20260927 02:36:00.844584 13480 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34871 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 02:36:00.844708 13480 heartbeater.cc:464] Registering TS with master...
I20260927 02:36:00.844875 13480 heartbeater.cc:511] Master 127.0.98.62:34871 requested a full tablet report, sending...
I20260927 02:36:00.845629 13480 heartbeater.cc:464] Registering TS with master...
I20260927 02:36:00.845789 13480 heartbeater.cc:511] Master 127.0.98.62:34871 requested a full tablet report, sending...
W20260927 02:36:00.846457 13480 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 02:36:00.992415 13483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:36:00.992650 13483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:36:00.992702 13483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:36:00.996673 13483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:36:00.996724 13483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:36:00.996795 13483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:36:01.002102 13483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:38227/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:38845
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38607
--tserver_master_addrs=127.0.98.62:34871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:36:01.003173 13483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:36:01.004115 13483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:36:01.013087 13489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:36:01.013092 13488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:36:01.013073 13491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:36:01.013818 13483 server_base.cc:1034] running on GCE node
I20260927 02:36:01.014264 13483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:36:01.014972 13483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:36:01.016175 13483 hybrid_clock.cc:648] HybridClock initialized: now 1790476561016155 us; error 32 us; skew 500 ppm
Sep 27 02:36:01 dist-test-slave-grlk krb5kdc[12150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:36:01.019800 13483 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:36:01.020898 13483 webserver.cc:460] Webserver started at http://127.0.98.3:38607/ using document root <none> and password file <none>
I20260927 02:36:01.021515 13483 fs_manager.cc:359] Metadata directory not provided
I20260927 02:36:01.021584 13483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 02:36:01 dist-test-slave-grlk krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:36:01.036764 13483 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 02:36:01.039470 13499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:01.040714 13483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 02:36:01.040814 13483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-2/wal
uuid: "930b2ebba1d444f19c396718c61d45c9"
format_stamp: "Formatted at 2026-09-27 02:35:59 on dist-test-slave-grlk"
server_key: "21059ae984ef48c25752a4678569ded8"
server_key_iv: "d6ad97408df9fff62e9de5991440d9b6"
server_key_version: "kuduclusterkey@0"
I20260927 02:36:01.041427 13483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:36:01.058403 13483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:36:01.061918 13483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:36:01.062131 13483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:36:01.062902 13483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:36:01.063939 13483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:36:01.063998 13483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:01.064057 13483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:36:01.064079 13483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:01.075678 13483 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38845
I20260927 02:36:01.075724 13612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38845 every 8 connection(s)
I20260927 02:36:01.077114 13483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:36:01.083885 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 13483
Sep 27 02:36:01 dist-test-slave-grlk krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:36:01.092732 13347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:01.079252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45451 (local address 127.0.98.62:34871)
0927 02:36:01.079387 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:01.079390 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:01.080590 (+ 1200us) server_negotiation.cc:407] Connection header received
0927 02:36:01.081586 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:01.081589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:01.081651 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:01.081733 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:36:01.082915 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:01.083463 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:01.085008 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:01.085308 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:01.085414 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:01.088288 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:36:01.088292 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:36:01.088294 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:36:01.088324 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:36:01.090066 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:01.090754 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:01.090757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:01.090759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:01.090870 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:01.091317 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:01.091321 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:01.091323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:01.091571 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:36:01.091678 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:01.092394 (+ 716us) server_negotiation.cc:299] Negotiation successful
0927 02:36:01.092553 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":53}
I20260927 02:36:01.093842 13615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:01.079758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34871 (local address 127.0.98.3:45451)
0927 02:36:01.080407 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:01.080460 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:01.081352 (+ 892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:01.081937 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:36:01.081944 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:36:01.082237 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:36:01.082753 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:01.082773 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:01.083585 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:01.083590 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:01.084806 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:01.084819 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:01.085530 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:01.085535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:01.085756 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:01.086582 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:36:01.086602 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:36:01.088098 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:36:01.090203 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:01.090210 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:01.090230 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:01.090624 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:01.090979 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:01.090982 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:01.090996 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:01.091190 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:01.091685 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:36:01.091692 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:36:01.092076 (+ 384us) client_negotiation.cc:769] Sending connection context
0927 02:36:01.092315 (+ 239us) client_negotiation.cc:240] Negotiation successful
0927 02:36:01.092686 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":148,"threads_started":1}
I20260927 02:36:01.095418 13613 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34871
I20260927 02:36:01.095753 13613 heartbeater.cc:464] Registering TS with master...
I20260927 02:36:01.096446 13613 heartbeater.cc:511] Master 127.0.98.62:34871 requested a full tablet report, sending...
W20260927 02:36:01.098415 13613 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34871 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 02:36:01.098578 13613 heartbeater.cc:464] Registering TS with master...
I20260927 02:36:01.098755 13613 heartbeater.cc:511] Master 127.0.98.62:34871 requested a full tablet report, sending...
I20260927 02:36:01.099637 13613 heartbeater.cc:464] Registering TS with master...
I20260927 02:36:01.099829 13613 heartbeater.cc:511] Master 127.0.98.62:34871 requested a full tablet report, sending...
I20260927 02:36:01.100800 13146 ts_manager.cc:194] Registered new tserver with Master: 930b2ebba1d444f19c396718c61d45c9 (127.0.98.3:38845)
I20260927 02:36:01.101390 13616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:01.084582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40076 (local address 127.0.98.62:34871)
0927 02:36:01.085146 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:01.085150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:01.085163 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:36:01.085208 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:01.085211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:01.085259 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:01.085357 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:36:01.086243 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:01.086996 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:01.088053 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:01.088378 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:01.088503 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:01.096196 (+ 7693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:36:01.096201 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:36:01.096203 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:36:01.096228 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:36:01.098050 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:01.098899 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:01.098905 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:01.098907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:01.099006 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:01.100176 (+ 1170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:01.100180 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:01.100182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:01.100423 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:36:01.100610 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:01.100900 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 02:36:01.101095 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":476,"thread_start_us":72,"threads_started":1}
I20260927 02:36:01.102489 13146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45451
Sep 27 02:36:01 dist-test-slave-grlk krb5kdc[12150](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790476561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:36:01.591426 13345 heartbeater.cc:464] Registering TS with master...
I20260927 02:36:01.591745 13345 heartbeater.cc:511] Master 127.0.98.62:34871 requested a full tablet report, sending...
I20260927 02:36:01.592547 13145 ts_manager.cc:194] Registered new tserver with Master: 5d2724da61d341379c6d243f088d3ed5 (127.0.98.1:44329)
I20260927 02:36:01.593389 13145 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54261
Sep 27 02:36:01 dist-test-slave-grlk krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:36:01.846665 13480 heartbeater.cc:464] Registering TS with master...
I20260927 02:36:01.846935 13480 heartbeater.cc:511] Master 127.0.98.62:34871 requested a full tablet report, sending...
I20260927 02:36:01.847566 13145 ts_manager.cc:194] Registered new tserver with Master: c269b7cd26b546dd8f92c51d5cfc98f6 (127.0.98.2:42501)
I20260927 02:36:01.848137 13145 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55151
I20260927 02:36:01.853417 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:36:01.864035 13635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:01.854808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40086 (local address 127.0.98.62:34871)
0927 02:36:01.855091 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:01.855095 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:01.855144 (+ 49us) server_negotiation.cc:407] Connection header received
0927 02:36:01.855275 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:01.855278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:01.855322 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:01.855392 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:36:01.856006 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:01.856527 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:01.857412 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:01.857681 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:01.857765 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:01.859129 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:36:01.859133 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:36:01.859136 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:36:01.859161 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:36:01.862218 (+ 3057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:01.862699 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:01.862702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:01.862704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:01.862789 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:01.863042 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:01.863045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:01.863047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:01.863252 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:36:01.863397 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:01.863592 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 02:36:01.863711 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":112,"threads_started":1}
I20260927 02:36:01.867795 13145 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40086:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:36:01.872215 13145 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:36:01.888115 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:01.882324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38845 (local address 127.0.0.1:43090)
0927 02:36:01.882644 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:01.882686 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:01.882793 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:01.883506 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:36:01.883512 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:36:01.883533 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:36:01.883657 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:01.883662 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:01.885087 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:01.885090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:01.886423 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:01.886433 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:01.887638 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:01.887641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:01.887748 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:01.887783 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 02:36:01.887876 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 02:36:01.887977 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":89,"threads_started":1}
I20260927 02:36:01.888713 13641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:01.882595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43090 (local address 127.0.98.3:38845)
0927 02:36:01.882935 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:01.882941 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:01.882963 (+ 22us) server_negotiation.cc:407] Connection header received
0927 02:36:01.883033 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:01.883038 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:01.883225 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:01.883360 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:36:01.883781 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:01.884968 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:01.886567 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:01.887347 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:01.887439 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:01.887581 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:01.888476 (+ 895us) server_negotiation.cc:299] Negotiation successful
0927 02:36:01.888585 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":79,"threads_started":1}
I20260927 02:36:01.891940 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:01.883616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56074 (local address 127.0.98.1:44329)
0927 02:36:01.884646 (+ 1030us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:01.884653 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:01.885458 (+ 805us) server_negotiation.cc:407] Connection header received
0927 02:36:01.885666 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:01.885672 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:01.885854 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:01.885963 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:36:01.886361 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:01.888190 (+ 1829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:01.890343 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:01.891157 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:01.891233 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:01.891357 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:01.891703 (+ 346us) server_negotiation.cc:299] Negotiation successful
0927 02:36:01.891803 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":921,"thread_start_us":78,"threads_started":1}
I20260927 02:36:01.891705 13547 tablet_service.cc:1467] Processing CreateTablet for tablet faa85d51e2fa458191949c064f7bd114 (DEFAULT_TABLE table=test-table [id=86137e5317824505b10593399daf1942]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:36:01.892333 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:01.883287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44329 (local address 127.0.0.1:56074)
0927 02:36:01.885359 (+ 2072us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:01.885370 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:01.885522 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:01.886129 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:36:01.886132 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:36:01.886148 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:36:01.886230 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:01.886236 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:01.888325 (+ 2089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:01.888328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:01.889528 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:01.889539 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:01.891460 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:01.891464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:01.891550 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:01.891573 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 02:36:01.892130 (+ 557us) client_negotiation.cc:240] Negotiation successful
0927 02:36:01.892209 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2018,"thread_start_us":65,"threads_started":1}
I20260927 02:36:01.892812 13547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faa85d51e2fa458191949c064f7bd114. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:36:01.894960 13278 tablet_service.cc:1467] Processing CreateTablet for tablet faa85d51e2fa458191949c064f7bd114 (DEFAULT_TABLE table=test-table [id=86137e5317824505b10593399daf1942]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:36:01.895869 13278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faa85d51e2fa458191949c064f7bd114. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:36:01.899483 13644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:01.882992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42501 (local address 127.0.0.1:41904)
0927 02:36:01.885103 (+ 2111us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:01.885116 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:01.885197 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:01.890249 (+ 5052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:36:01.890252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:36:01.890268 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:36:01.890346 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:01.890351 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:01.891774 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:01.891777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:01.897186 (+ 5409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:01.897197 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:01.899050 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:01.899054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:01.899150 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:01.899217 (+ 67us) client_negotiation.cc:769] Sending connection context
0927 02:36:01.899269 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 02:36:01.899339 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2051,"thread_start_us":85,"threads_started":1}
I20260927 02:36:01.900336 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:01.883142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41904 (local address 127.0.98.2:42501)
0927 02:36:01.889835 (+ 6693us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:01.889842 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:01.889865 (+ 23us) server_negotiation.cc:407] Connection header received
0927 02:36:01.889931 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:01.889936 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:01.890107 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:01.890478 (+ 371us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:36:01.890629 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:01.891646 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:01.897329 (+ 5683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:01.898919 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:01.899790 (+ 871us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:01.899975 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:01.900028 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 02:36:01.900120 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6592,"thread_start_us":110,"threads_started":1}
I20260927 02:36:01.903024 13404 tablet_service.cc:1467] Processing CreateTablet for tablet faa85d51e2fa458191949c064f7bd114 (DEFAULT_TABLE table=test-table [id=86137e5317824505b10593399daf1942]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:36:01.903631 13647 tablet_bootstrap.cc:492] T faa85d51e2fa458191949c064f7bd114 P 5d2724da61d341379c6d243f088d3ed5: Bootstrap starting.
I20260927 02:36:01.903977 13404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faa85d51e2fa458191949c064f7bd114. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:36:01.905750 13647 tablet_bootstrap.cc:654] T faa85d51e2fa458191949c064f7bd114 P 5d2724da61d341379c6d243f088d3ed5: Neither blocks nor log segments found. Creating new log.
I20260927 02:36:01.906526 13647 log.cc:824] T faa85d51e2fa458191949c064f7bd114 P 5d2724da61d341379c6d243f088d3ed5: Log is configured to *not* fsync() on all Append() calls
I20260927 02:36:01.908365 13647 tablet_bootstrap.cc:492] T faa85d51e2fa458191949c064f7bd114 P 5d2724da61d341379c6d243f088d3ed5: No bootstrap required, opened a new log
I20260927 02:36:01.908525 13647 ts_tablet_manager.cc:1397] T faa85d51e2fa458191949c064f7bd114 P 5d2724da61d341379c6d243f088d3ed5: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 02:36:01.909224 13646 tablet_bootstrap.cc:492] T faa85d51e2fa458191949c064f7bd114 P 930b2ebba1d444f19c396718c61d45c9: Bootstrap starting.
I20260927 02:36:01.911226 13646 tablet_bootstrap.cc:654] T faa85d51e2fa458191949c064f7bd114 P 930b2ebba1d444f19c396718c61d45c9: Neither blocks nor log segments found. Creating new log.
I20260927 02:36:01.911993 13646 log.cc:824] T faa85d51e2fa458191949c064f7bd114 P 930b2ebba1d444f19c396718c61d45c9: Log is configured to *not* fsync() on all Append() calls
I20260927 02:36:01.912437 13647 raft_consensus.cc:348] T faa85d51e2fa458191949c064f7bd114 P 5d2724da61d341379c6d243f088d3ed5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "930b2ebba1d444f19c396718c61d45c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38845 } } peers { permanent_uuid: "c269b7cd26b546dd8f92c51d5cfc98f6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42501 } } peers { permanent_uuid: "5d2724da61d341379c6d243f088d3ed5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44329 } }
I20260927 02:36:01.912683 13647 raft_consensus.cc:374] T faa85d51e2fa458191949c064f7bd114 P 5d2724da61d341379c6d243f088d3ed5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:36:01.912731 13647 raft_consensus.cc:729] T faa85d51e2fa458191949c064f7bd114 P 5d2724da61d341379c6d243f088d3ed5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d2724da61d341379c6d243f088d3ed5, State: Initialized, Role: FOLLOWER
I20260927 02:36:01.913205 13649 tablet_bootstrap.cc:492] T faa85d51e2fa458191949c064f7bd114 P c269b7cd26b546dd8f92c51d5cfc98f6: Bootstrap starting.
I20260927 02:36:01.913193 13647 consensus_queue.cc:260] T faa85d51e2fa458191949c064f7bd114 P 5d2724da61d341379c6d243f088d3ed5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "930b2ebba1d444f19c396718c61d45c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38845 } } peers { permanent_uuid: "c269b7cd26b546dd8f92c51d5cfc98f6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42501 } } peers { permanent_uuid: "5d2724da61d341379c6d243f088d3ed5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44329 } }
I20260927 02:36:01.914129 13647 ts_tablet_manager.cc:1428] T faa85d51e2fa458191949c064f7bd114 P 5d2724da61d341379c6d243f088d3ed5: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 02:36:01.914443 13345 heartbeater.cc:503] Master 127.0.98.62:34871 was elected leader, sending a full tablet report...
I20260927 02:36:01.915367 13649 tablet_bootstrap.cc:654] T faa85d51e2fa458191949c064f7bd114 P c269b7cd26b546dd8f92c51d5cfc98f6: Neither blocks nor log segments found. Creating new log.
I20260927 02:36:01.917693 13649 log.cc:824] T faa85d51e2fa458191949c064f7bd114 P c269b7cd26b546dd8f92c51d5cfc98f6: Log is configured to *not* fsync() on all Append() calls
I20260927 02:36:01.918836 13646 tablet_bootstrap.cc:492] T faa85d51e2fa458191949c064f7bd114 P 930b2ebba1d444f19c396718c61d45c9: No bootstrap required, opened a new log
I20260927 02:36:01.919025 13646 ts_tablet_manager.cc:1397] T faa85d51e2fa458191949c064f7bd114 P 930b2ebba1d444f19c396718c61d45c9: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260927 02:36:01.919499 13649 tablet_bootstrap.cc:492] T faa85d51e2fa458191949c064f7bd114 P c269b7cd26b546dd8f92c51d5cfc98f6: No bootstrap required, opened a new log
I20260927 02:36:01.919649 13649 ts_tablet_manager.cc:1397] T faa85d51e2fa458191949c064f7bd114 P c269b7cd26b546dd8f92c51d5cfc98f6: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260927 02:36:01.923303 13646 raft_consensus.cc:348] T faa85d51e2fa458191949c064f7bd114 P 930b2ebba1d444f19c396718c61d45c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "930b2ebba1d444f19c396718c61d45c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38845 } } peers { permanent_uuid: "c269b7cd26b546dd8f92c51d5cfc98f6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42501 } } peers { permanent_uuid: "5d2724da61d341379c6d243f088d3ed5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44329 } }
I20260927 02:36:01.923413 13649 raft_consensus.cc:348] T faa85d51e2fa458191949c064f7bd114 P c269b7cd26b546dd8f92c51d5cfc98f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "930b2ebba1d444f19c396718c61d45c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38845 } } peers { permanent_uuid: "c269b7cd26b546dd8f92c51d5cfc98f6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42501 } } peers { permanent_uuid: "5d2724da61d341379c6d243f088d3ed5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44329 } }
I20260927 02:36:01.923544 13646 raft_consensus.cc:374] T faa85d51e2fa458191949c064f7bd114 P 930b2ebba1d444f19c396718c61d45c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:36:01.923592 13649 raft_consensus.cc:374] T faa85d51e2fa458191949c064f7bd114 P c269b7cd26b546dd8f92c51d5cfc98f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:36:01.923595 13646 raft_consensus.cc:729] T faa85d51e2fa458191949c064f7bd114 P 930b2ebba1d444f19c396718c61d45c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 930b2ebba1d444f19c396718c61d45c9, State: Initialized, Role: FOLLOWER
I20260927 02:36:01.923631 13649 raft_consensus.cc:729] T faa85d51e2fa458191949c064f7bd114 P c269b7cd26b546dd8f92c51d5cfc98f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c269b7cd26b546dd8f92c51d5cfc98f6, State: Initialized, Role: FOLLOWER
I20260927 02:36:01.924048 13646 consensus_queue.cc:260] T faa85d51e2fa458191949c064f7bd114 P 930b2ebba1d444f19c396718c61d45c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "930b2ebba1d444f19c396718c61d45c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38845 } } peers { permanent_uuid: "c269b7cd26b546dd8f92c51d5cfc98f6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42501 } } peers { permanent_uuid: "5d2724da61d341379c6d243f088d3ed5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44329 } }
I20260927 02:36:01.925045 13646 ts_tablet_manager.cc:1428] T faa85d51e2fa458191949c064f7bd114 P 930b2ebba1d444f19c396718c61d45c9: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 02:36:01.925052 13649 consensus_queue.cc:260] T faa85d51e2fa458191949c064f7bd114 P c269b7cd26b546dd8f92c51d5cfc98f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "930b2ebba1d444f19c396718c61d45c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38845 } } peers { permanent_uuid: "c269b7cd26b546dd8f92c51d5cfc98f6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42501 } } peers { permanent_uuid: "5d2724da61d341379c6d243f088d3ed5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44329 } }
I20260927 02:36:01.925362 13613 heartbeater.cc:503] Master 127.0.98.62:34871 was elected leader, sending a full tablet report...
I20260927 02:36:01.926044 13649 ts_tablet_manager.cc:1428] T faa85d51e2fa458191949c064f7bd114 P c269b7cd26b546dd8f92c51d5cfc98f6: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 02:36:01.926359 13480 heartbeater.cc:503] Master 127.0.98.62:34871 was elected leader, sending a full tablet report...
Sep 27 02:36:02 dist-test-slave-grlk krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260927 02:36:02.056252 13346 tablet.cc:2369] T faa85d51e2fa458191949c064f7bd114 P 5d2724da61d341379c6d243f088d3ed5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:36:02.074676 13481 tablet.cc:2369] T faa85d51e2fa458191949c064f7bd114 P c269b7cd26b546dd8f92c51d5cfc98f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:36:02.078729 13614 tablet.cc:2369] T faa85d51e2fa458191949c064f7bd114 P 930b2ebba1d444f19c396718c61d45c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:36:02.083652 13654 raft_consensus.cc:482] T faa85d51e2fa458191949c064f7bd114 P c269b7cd26b546dd8f92c51d5cfc98f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:36:02.083844 13654 raft_consensus.cc:504] T faa85d51e2fa458191949c064f7bd114 P c269b7cd26b546dd8f92c51d5cfc98f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "930b2ebba1d444f19c396718c61d45c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38845 } } peers { permanent_uuid: "c269b7cd26b546dd8f92c51d5cfc98f6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42501 } } peers { permanent_uuid: "5d2724da61d341379c6d243f088d3ed5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44329 } }
I20260927 02:36:02.085142 13654 leader_election.cc:288] T faa85d51e2fa458191949c064f7bd114 P c269b7cd26b546dd8f92c51d5cfc98f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 930b2ebba1d444f19c396718c61d45c9 (127.0.98.3:38845), 5d2724da61d341379c6d243f088d3ed5 (127.0.98.1:44329)
I20260927 02:36:02.092976 13641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:02.086193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53543 (local address 127.0.98.3:38845)
0927 02:36:02.086310 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:02.086314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:02.087333 (+ 1019us) server_negotiation.cc:407] Connection header received
0927 02:36:02.087560 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:02.087564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:02.087609 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:02.087654 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:36:02.088349 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:02.089163 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:02.091281 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:02.091912 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:02.091997 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:02.092072 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:02.092723 (+ 651us) server_negotiation.cc:299] Negotiation successful
0927 02:36:02.092813 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 02:36:02.093652 13657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:02.085489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38845 (local address 127.0.98.2:53543)
0927 02:36:02.087242 (+ 1753us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:02.087257 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:02.087413 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:02.087918 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:36:02.087921 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:36:02.087940 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:36:02.088217 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:02.088225 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:02.089306 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:02.089309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:02.091127 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:02.091139 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:02.092430 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:02.092435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:02.092565 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:02.092591 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 02:36:02.093405 (+ 814us) client_negotiation.cc:240] Negotiation successful
0927 02:36:02.093504 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1677,"thread_start_us":109,"threads_started":1}
I20260927 02:36:02.094702 13567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa85d51e2fa458191949c064f7bd114" candidate_uuid: "c269b7cd26b546dd8f92c51d5cfc98f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "930b2ebba1d444f19c396718c61d45c9" is_pre_election: true
I20260927 02:36:02.095062 13567 raft_consensus.cc:2454] T faa85d51e2fa458191949c064f7bd114 P 930b2ebba1d444f19c396718c61d45c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c269b7cd26b546dd8f92c51d5cfc98f6 in term 0.
I20260927 02:36:02.095701 13366 leader_election.cc:302] T faa85d51e2fa458191949c064f7bd114 P c269b7cd26b546dd8f92c51d5cfc98f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 930b2ebba1d444f19c396718c61d45c9, c269b7cd26b546dd8f92c51d5cfc98f6; no voters:
I20260927 02:36:02.096020 13654 raft_consensus.cc:2781] T faa85d51e2fa458191949c064f7bd114 P c269b7cd26b546dd8f92c51d5cfc98f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:36:02.096092 13654 raft_consensus.cc:482] T faa85d51e2fa458191949c064f7bd114 P c269b7cd26b546dd8f92c51d5cfc98f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:36:02.096129 13654 raft_consensus.cc:3037] T faa85d51e2fa458191949c064f7bd114 P c269b7cd26b546dd8f92c51d5cfc98f6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:36:02.097471 13654 raft_consensus.cc:504] T faa85d51e2fa458191949c064f7bd114 P c269b7cd26b546dd8f92c51d5cfc98f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "930b2ebba1d444f19c396718c61d45c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38845 } } peers { permanent_uuid: "c269b7cd26b546dd8f92c51d5cfc98f6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42501 } } peers { permanent_uuid: "5d2724da61d341379c6d243f088d3ed5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44329 } }
I20260927 02:36:02.097853 13654 leader_election.cc:288] T faa85d51e2fa458191949c064f7bd114 P c269b7cd26b546dd8f92c51d5cfc98f6 [CANDIDATE]: Term 1 election: Requested vote from peers 930b2ebba1d444f19c396718c61d45c9 (127.0.98.3:38845), 5d2724da61d341379c6d243f088d3ed5 (127.0.98.1:44329)
I20260927 02:36:02.098562 13567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa85d51e2fa458191949c064f7bd114" candidate_uuid: "c269b7cd26b546dd8f92c51d5cfc98f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "930b2ebba1d444f19c396718c61d45c9"
I20260927 02:36:02.098673 13567 raft_consensus.cc:3037] T faa85d51e2fa458191949c064f7bd114 P 930b2ebba1d444f19c396718c61d45c9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:36:02.099799 13567 raft_consensus.cc:2454] T faa85d51e2fa458191949c064f7bd114 P 930b2ebba1d444f19c396718c61d45c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c269b7cd26b546dd8f92c51d5cfc98f6 in term 1.
I20260927 02:36:02.100211 13366 leader_election.cc:302] T faa85d51e2fa458191949c064f7bd114 P c269b7cd26b546dd8f92c51d5cfc98f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 930b2ebba1d444f19c396718c61d45c9, c269b7cd26b546dd8f92c51d5cfc98f6; no voters:
I20260927 02:36:02.100399 13654 raft_consensus.cc:2781] T faa85d51e2fa458191949c064f7bd114 P c269b7cd26b546dd8f92c51d5cfc98f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:36:02.100584 13654 raft_consensus.cc:686] T faa85d51e2fa458191949c064f7bd114 P c269b7cd26b546dd8f92c51d5cfc98f6 [term 1 LEADER]: Becoming Leader. State: Replica: c269b7cd26b546dd8f92c51d5cfc98f6, State: Running, Role: LEADER
I20260927 02:36:02.100919 13654 consensus_queue.cc:237] T faa85d51e2fa458191949c064f7bd114 P c269b7cd26b546dd8f92c51d5cfc98f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "930b2ebba1d444f19c396718c61d45c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38845 } } peers { permanent_uuid: "c269b7cd26b546dd8f92c51d5cfc98f6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42501 } } peers { permanent_uuid: "5d2724da61d341379c6d243f088d3ed5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44329 } }
I20260927 02:36:02.101392 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:02.085811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44329 (local address 127.0.98.2:60791)
0927 02:36:02.097113 (+ 11302us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:02.097128 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:02.097236 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:02.097582 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:36:02.097586 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:36:02.097604 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:36:02.097708 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:02.097714 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:02.098778 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:02.098781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:02.099963 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:02.099974 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:02.100976 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:02.100980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:02.101088 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:02.101112 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 02:36:02.101181 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 02:36:02.101244 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11231,"thread_start_us":67,"threads_started":1}
I20260927 02:36:02.101783 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:02.086297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60791 (local address 127.0.98.1:44329)
0927 02:36:02.086561 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:02.086565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:02.097227 (+ 10662us) server_negotiation.cc:407] Connection header received
0927 02:36:02.097330 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:02.097333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:02.097390 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:02.097467 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:36:02.097834 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:02.098649 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:02.100112 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:02.100730 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:02.100828 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:02.100890 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:02.101582 (+ 692us) server_negotiation.cc:299] Negotiation successful
0927 02:36:02.101664 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179}
I20260927 02:36:02.102794 13294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa85d51e2fa458191949c064f7bd114" candidate_uuid: "c269b7cd26b546dd8f92c51d5cfc98f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d2724da61d341379c6d243f088d3ed5" is_pre_election: true
I20260927 02:36:02.103099 13294 raft_consensus.cc:2454] T faa85d51e2fa458191949c064f7bd114 P 5d2724da61d341379c6d243f088d3ed5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c269b7cd26b546dd8f92c51d5cfc98f6 in term 0.
I20260927 02:36:02.103420 13295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa85d51e2fa458191949c064f7bd114" candidate_uuid: "c269b7cd26b546dd8f92c51d5cfc98f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d2724da61d341379c6d243f088d3ed5"
I20260927 02:36:02.103539 13295 raft_consensus.cc:3037] T faa85d51e2fa458191949c064f7bd114 P 5d2724da61d341379c6d243f088d3ed5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:36:02.104772 13295 raft_consensus.cc:2454] T faa85d51e2fa458191949c064f7bd114 P 5d2724da61d341379c6d243f088d3ed5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c269b7cd26b546dd8f92c51d5cfc98f6 in term 1.
I20260927 02:36:02.105618 13145 catalog_manager.cc:5495] T faa85d51e2fa458191949c064f7bd114 P c269b7cd26b546dd8f92c51d5cfc98f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to c269b7cd26b546dd8f92c51d5cfc98f6 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "c269b7cd26b546dd8f92c51d5cfc98f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "930b2ebba1d444f19c396718c61d45c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c269b7cd26b546dd8f92c51d5cfc98f6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d2724da61d341379c6d243f088d3ed5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44329 } health_report { overall_health: UNKNOWN } } }
I20260927 02:36:02.129801 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:02.124782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41914 (local address 127.0.98.2:42501)
0927 02:36:02.124889 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:02.124894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:02.124907 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:36:02.125008 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:02.125011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:02.125082 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:02.125192 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:36:02.125455 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:02.126256 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:02.127648 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:02.127966 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:02.128074 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:02.128479 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:36:02.129090 (+ 611us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:36:02.129194 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:02.129586 (+ 392us) server_negotiation.cc:299] Negotiation successful
0927 02:36:02.129645 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 02:36:02.136750 13567 raft_consensus.cc:1261] T faa85d51e2fa458191949c064f7bd114 P 930b2ebba1d444f19c396718c61d45c9 [term 1 FOLLOWER]: Refusing update from remote peer c269b7cd26b546dd8f92c51d5cfc98f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:36:02.136750 13295 raft_consensus.cc:1261] T faa85d51e2fa458191949c064f7bd114 P 5d2724da61d341379c6d243f088d3ed5 [term 1 FOLLOWER]: Refusing update from remote peer c269b7cd26b546dd8f92c51d5cfc98f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:36:02.137547 13654 consensus_queue.cc:1035] T faa85d51e2fa458191949c064f7bd114 P c269b7cd26b546dd8f92c51d5cfc98f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d2724da61d341379c6d243f088d3ed5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:36:02.138000 13654 consensus_queue.cc:1035] T faa85d51e2fa458191949c064f7bd114 P c269b7cd26b546dd8f92c51d5cfc98f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "930b2ebba1d444f19c396718c61d45c9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:36:02.147395 13662 mvcc.cc:205] Tried to move back new op lower bound from 7333791998504132608 to 7333791998368124928. Current Snapshot: MvccSnapshot[applied={T|T < 7333791998504132608}]
I20260927 02:36:02.150354 13664 mvcc.cc:205] Tried to move back new op lower bound from 7333791998504132608 to 7333791998368124928. Current Snapshot: MvccSnapshot[applied={T|T < 7333791998504132608}]
I20260927 02:36:02.159572 13146 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:36:02.159750 13146 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:36:02.162433 13146 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e35e5f8d6bec46c2a790209417aec057: Sending DeleteTablet for 3 replicas of tablet faa85d51e2fa458191949c064f7bd114
I20260927 02:36:02.163443 13404 tablet_service.cc:1514] Processing DeleteTablet for tablet faa85d51e2fa458191949c064f7bd114 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:36:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41904
I20260927 02:36:02.163561 13547 tablet_service.cc:1514] Processing DeleteTablet for tablet faa85d51e2fa458191949c064f7bd114 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:36:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43090
I20260927 02:36:02.163961 13278 tablet_service.cc:1514] Processing DeleteTablet for tablet faa85d51e2fa458191949c064f7bd114 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:36:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56074
I20260927 02:36:02.164149 13671 tablet_replica.cc:330] T faa85d51e2fa458191949c064f7bd114 P 930b2ebba1d444f19c396718c61d45c9: stopping tablet replica
I20260927 02:36:02.164541 13671 raft_consensus.cc:2229] T faa85d51e2fa458191949c064f7bd114 P 930b2ebba1d444f19c396718c61d45c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:36:02.164793 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 13184
I20260927 02:36:02.167554 13671 pending_rounds.cc:62] T faa85d51e2fa458191949c064f7bd114 P 930b2ebba1d444f19c396718c61d45c9: Trying to abort 2 pending ops.
I20260927 02:36:02.167709 13671 pending_rounds.cc:68] T faa85d51e2fa458191949c064f7bd114 P 930b2ebba1d444f19c396718c61d45c9: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7333791998504132608 op_type: WRITE_OP write_request { tablet_id: "faa85d51e2fa458191949c064f7bd114" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\276\204\342\325\006\"8\n\ntest-admin\022*\n 86137e5317824505b10593399daf1942\020\001\030\001 \001(\001" signature: "<redacted>""\000\342\305\340\302\023_z\005\366j\330\010\340\201^\350\351\243\304.\256@,\033\225\362h\301\243\177G\204\351Y`o\313\356\016/\307|ewA\332\346H)\251\177\301\274Gtp\021\324\304\222W}u" signing_key_seq_num: 0 } } request_id { client_id: "5706855c847b46c0a4e38cdb9a1c83bb" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260927 02:36:02.168228 13671 pending_rounds.cc:68] T faa85d51e2fa458191949c064f7bd114 P 930b2ebba1d444f19c396718c61d45c9: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7333791998368124928 op_type: NO_OP noop_request { }
I20260927 02:36:02.168398 13671 raft_consensus.cc:2866] T faa85d51e2fa458191949c064f7bd114 P 930b2ebba1d444f19c396718c61d45c9 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260927 02:36:02.168527 13671 raft_consensus.cc:2258] T faa85d51e2fa458191949c064f7bd114 P 930b2ebba1d444f19c396718c61d45c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:36:02.173549 13670 tablet_replica.cc:330] T faa85d51e2fa458191949c064f7bd114 P c269b7cd26b546dd8f92c51d5cfc98f6: stopping tablet replica
I20260927 02:36:02.173926 13670 raft_consensus.cc:2229] T faa85d51e2fa458191949c064f7bd114 P c269b7cd26b546dd8f92c51d5cfc98f6 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:36:02.174379 13670 raft_consensus.cc:2258] T faa85d51e2fa458191949c064f7bd114 P c269b7cd26b546dd8f92c51d5cfc98f6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 02:36:02.174846 13130 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44329 (error 108)
W20260927 02:36:02.176050 13665 consensus_queue.cc:1159] T faa85d51e2fa458191949c064f7bd114 P 930b2ebba1d444f19c396718c61d45c9 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "930b2ebba1d444f19c396718c61d45c9" status { last_received { term: 1 index: 2 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 2 } }
I20260927 02:36:02.176108 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 13349
I20260927 02:36:02.179811 13644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:02.179494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44329 (local address 127.0.0.1:56086)
0927 02:36:02.179666 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:02.179726 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260927 02:36:02.180056 13130 catalog_manager.cc:4584] TS 5d2724da61d341379c6d243f088d3ed5 (127.0.98.1:44329): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet faa85d51e2fa458191949c064f7bd114: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44329: connect: Connection refused (error 111)
I20260927 02:36:02.187680 13671 ts_tablet_manager.cc:1905] T faa85d51e2fa458191949c064f7bd114 P 930b2ebba1d444f19c396718c61d45c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:36:02.190559 13671 ts_tablet_manager.cc:1918] T faa85d51e2fa458191949c064f7bd114 P 930b2ebba1d444f19c396718c61d45c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:36:02.190706 13671 log.cc:1196] T faa85d51e2fa458191949c064f7bd114 P 930b2ebba1d444f19c396718c61d45c9: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476410933508-392-0/minicluster-data/ts-2/wal/wals/faa85d51e2fa458191949c064f7bd114
I20260927 02:36:02.191058 13671 ts_tablet_manager.cc:1939] T faa85d51e2fa458191949c064f7bd114 P 930b2ebba1d444f19c396718c61d45c9: Deleting consensus metadata
W20260927 02:36:02.191187 13501 connection.cc:535] server connection from 127.0.98.2:53543 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 02:36:02.192502 13130 catalog_manager.cc:4841] TS 930b2ebba1d444f19c396718c61d45c9 (127.0.98.3:38845): tablet faa85d51e2fa458191949c064f7bd114 (table test-table [id=86137e5317824505b10593399daf1942]) successfully deleted
I20260927 02:36:02.193051 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 13483
I20260927 02:36:02.194591 13644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:02.194302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42501 (local address 127.0.0.1:41926)
0927 02:36:02.194464 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:02.194513 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260927 02:36:02.204552 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 13113
I20260927 02:36:02.209944 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 02:36:02.544633 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 02:36:02.544761 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 02:36:02.879384 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 02:36:02.879 UTC [12180] LOG: received smart shutdown request
2026-09-27 02:36:02.883 UTC [12180] LOG: background worker "logical replication launcher" (PID 12189) exited with exit code 1
2026-09-27 02:36:02.883 UTC [12184] LOG: shutting down
2026-09-27 02:36:03.032 UTC [12180] LOG: database system is shut down
2026-09-27T02:36:03Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (45403 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:36:03 dist-test-slave-grlk krb5kdc[13682](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:36:03 dist-test-slave-grlk krb5kdc[13682](info): set up 2 sockets
Sep 27 02:36:03 dist-test-slave-grlk krb5kdc[13682](info): commencing operation
krb5kdc: starting...
W20260927 02:36:04.092633 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/krb5kdc/test-user.keytab.
Sep 27 02:36:04 dist-test-slave-grlk krb5kdc[13682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:36:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:36:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:36:04.207996 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33061
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33061,127.0.98.61:43729,127.0.98.60:43891
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:36:04.315332 13695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:36:04.315637 13695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:36:04.315702 13695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:36:04.319881 13695 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:36:04.319953 13695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:36:04.319984 13695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:36:04.320005 13695 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:36:04.320026 13695 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:36:04.325368 13695 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33061,127.0.98.61:43729,127.0.98.60:43891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:36:04.326339 13695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:36:04.327289 13695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:36:04.335181 13703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:36:04.335407 13695 server_base.cc:1034] running on GCE node
W20260927 02:36:04.335181 13700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:36:04.335222 13701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:36:04.336018 13695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:36:04.337064 13695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:36:04.338217 13695 hybrid_clock.cc:648] HybridClock initialized: now 1790476564338199 us; error 35 us; skew 500 ppm
Sep 27 02:36:04 dist-test-slave-grlk krb5kdc[13682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:36:04.344875 13695 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:36:04.345845 13695 webserver.cc:460] Webserver started at http://127.0.98.62:40309/ using document root <none> and password file <none>
I20260927 02:36:04.346388 13695 fs_manager.cc:359] Metadata directory not provided
I20260927 02:36:04.346458 13695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:36:04.346652 13695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:36:04.348649 13695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "fde097f4a50045cabbc75590872a17f3"
format_stamp: "Formatted at 2026-09-27 02:36:04 on dist-test-slave-grlk"
server_key: "89c8e65a0118250b39bd4a96c77dd390"
server_key_iv: "14244afefac81e508d7ca57d204f313b"
server_key_version: "encryptionkey@0"
I20260927 02:36:04.349216 13695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "fde097f4a50045cabbc75590872a17f3"
format_stamp: "Formatted at 2026-09-27 02:36:04 on dist-test-slave-grlk"
server_key: "89c8e65a0118250b39bd4a96c77dd390"
server_key_iv: "14244afefac81e508d7ca57d204f313b"
server_key_version: "encryptionkey@0"
I20260927 02:36:04.352638 13695 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:36:04.354982 13710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:04.356107 13695 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:36:04.356216 13695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "fde097f4a50045cabbc75590872a17f3"
format_stamp: "Formatted at 2026-09-27 02:36:04 on dist-test-slave-grlk"
server_key: "89c8e65a0118250b39bd4a96c77dd390"
server_key_iv: "14244afefac81e508d7ca57d204f313b"
server_key_version: "encryptionkey@0"
I20260927 02:36:04.356303 13695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:36:04.374310 13695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:36:04.376782 13695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:36:04.376973 13695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:36:04.383548 13695 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33061
I20260927 02:36:04.383565 13762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33061 every 8 connection(s)
I20260927 02:36:04.384415 13695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:36:04.386940 13763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:36:04.390645 13763 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33061 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:36:04.393702 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 13695
I20260927 02:36:04.393831 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:36:04.394111 392 external_mini_cluster.cc:1428] Setting key a3e2cc702b320f21139760bced57f9ba
Sep 27 02:36:04 dist-test-slave-grlk krb5kdc[13682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260927 02:36:04.405558 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:04.392969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39724 (local address 127.0.98.62:33061)
0927 02:36:04.393420 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:04.393455 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:04.393760 (+ 305us) server_negotiation.cc:407] Connection header received
0927 02:36:04.394796 (+ 1036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:04.394800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:04.394982 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:04.395136 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:36:04.395921 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:04.397129 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:04.398125 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:04.398695 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:04.398855 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:04.401160 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:36:04.401167 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:36:04.401178 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:36:04.401202 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:36:04.402658 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:04.403115 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:04.403120 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:04.403122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:04.403242 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:04.403541 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:04.403544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:04.403546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:04.403914 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:36:04.404029 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:04.404337 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 02:36:04.404545 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":309,"spinlock_wait_cycles":50944,"thread_start_us":149,"threads_started":1}
I20260927 02:36:04.405558 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:04.392969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.0.1:39724)
0927 02:36:04.393624 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:04.393659 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:04.394521 (+ 862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:04.395144 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:36:04.395149 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:36:04.395558 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:36:04.395782 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:04.395794 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:04.397243 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:04.397246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:04.398010 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:04.398020 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:04.398824 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:04.398826 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:04.398945 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:04.399485 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:36:04.399501 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:36:04.401033 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:36:04.402769 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:04.402780 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:04.402787 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:04.403012 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:04.403345 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:04.403348 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:04.403350 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:04.403460 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:04.404012 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:36:04.404017 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:36:04.404133 (+ 116us) client_negotiation.cc:769] Sending connection context
0927 02:36:04.404271 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 02:36:04.404606 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"mutex_wait_us":81,"thread_start_us":150,"threads_started":1}
I20260927 02:36:04.407877 13763 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43729 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:36:04.408670 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:04.408376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43729 (local address 127.0.0.1:53876)
0927 02:36:04.408523 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:04.408604 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 02:36:04.408842 13711 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43729: connect: Connection refused (error 111)
I20260927 02:36:04.409647 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:04.409427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43729 (local address 127.0.0.1:53882)
0927 02:36:04.409535 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:04.409585 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 02:36:04.409791 13763 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43729: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43729: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 02:36:04.417088 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43729
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33061,127.0.98.61:43729,127.0.98.60:43891
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:36:04.458997 13763 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43729 } attempt: 1
I20260927 02:36:04.459849 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:04.459543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43729 (local address 127.0.0.1:53894)
0927 02:36:04.459704 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:04.459772 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 02:36:04.460523 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:04.460301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43729 (local address 127.0.0.1:53898)
0927 02:36:04.460424 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:04.460468 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 02:36:04.460682 13763 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43729: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43729: connect: Connection refused (error 111)
W20260927 02:36:04.520421 13770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:36:04.520655 13770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:36:04.520692 13770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:36:04.524530 13770 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:36:04.524586 13770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:36:04.524603 13770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:36:04.524617 13770 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:36:04.524636 13770 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:36:04.528885 13763 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43729 } attempt: 2
I20260927 02:36:04.529709 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:04.529427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43729 (local address 127.0.0.1:53910)
0927 02:36:04.529573 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:04.529631 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 02:36:04.529703 13770 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33061,127.0.98.61:43729,127.0.98.60:43891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:36:04.530407 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:04.530209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43729 (local address 127.0.0.1:53914)
0927 02:36:04.530307 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:04.530352 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 02:36:04.530544 13763 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43729: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43729: connect: Connection refused (error 111)
I20260927 02:36:04.530768 13770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:36:04.531687 13770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:36:04.539364 13776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:36:04.539371 13778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:36:04.539450 13775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:36:04.539659 13770 server_base.cc:1034] running on GCE node
I20260927 02:36:04.540146 13770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:36:04.541174 13770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:36:04.542322 13770 hybrid_clock.cc:648] HybridClock initialized: now 1790476564542303 us; error 29 us; skew 500 ppm
Sep 27 02:36:04 dist-test-slave-grlk krb5kdc[13682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:36:04.544728 13770 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 02:36:04.545537 13770 webserver.cc:460] Webserver started at http://127.0.98.61:35783/ using document root <none> and password file <none>
I20260927 02:36:04.546027 13770 fs_manager.cc:359] Metadata directory not provided
I20260927 02:36:04.546096 13770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:36:04.546264 13770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:36:04.548108 13770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/minicluster-data/master-1/data/instance:
uuid: "0bd26fd6f4bb454abb0a1b8da25bd3ab"
format_stamp: "Formatted at 2026-09-27 02:36:04 on dist-test-slave-grlk"
server_key: "f4202ba77fcc2731c5763051b8e66a79"
server_key_iv: "fe3d2ff79403faf234aa6789ce0cf08e"
server_key_version: "encryptionkey@0"
I20260927 02:36:04.548589 13770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/minicluster-data/master-1/wal/instance:
uuid: "0bd26fd6f4bb454abb0a1b8da25bd3ab"
format_stamp: "Formatted at 2026-09-27 02:36:04 on dist-test-slave-grlk"
server_key: "f4202ba77fcc2731c5763051b8e66a79"
server_key_iv: "fe3d2ff79403faf234aa6789ce0cf08e"
server_key_version: "encryptionkey@0"
I20260927 02:36:04.551860 13770 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 02:36:04.554044 13785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:04.555238 13770 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:36:04.555361 13770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/minicluster-data/master-1/wal
uuid: "0bd26fd6f4bb454abb0a1b8da25bd3ab"
format_stamp: "Formatted at 2026-09-27 02:36:04 on dist-test-slave-grlk"
server_key: "f4202ba77fcc2731c5763051b8e66a79"
server_key_iv: "fe3d2ff79403faf234aa6789ce0cf08e"
server_key_version: "encryptionkey@0"
I20260927 02:36:04.555461 13770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:36:04.589651 13770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:36:04.592473 13770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:36:04.592646 13770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:36:04.599833 13770 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43729
I20260927 02:36:04.599844 13837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43729 every 8 connection(s)
I20260927 02:36:04.600762 13770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/minicluster-data/master-1/data/info.pb
I20260927 02:36:04.602290 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 13770
I20260927 02:36:04.602433 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/minicluster-data/master-1/wal/instance
I20260927 02:36:04.602734 392 external_mini_cluster.cc:1428] Setting key de0a018d55e60d1bef5c1a7b92cc4053
I20260927 02:36:04.603843 13838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:36:04.607739 13838 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33061 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 02:36:04 dist-test-slave-grlk krb5kdc[13682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:36:04.621675 13763 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43729 } attempt: 3
I20260927 02:36:04.621969 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:04.609924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39738 (local address 127.0.98.62:33061)
0927 02:36:04.610099 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:04.610103 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:04.610921 (+ 818us) server_negotiation.cc:407] Connection header received
0927 02:36:04.611937 (+ 1016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:04.611940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:04.611989 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:04.612089 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:36:04.613264 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:04.613792 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:04.615373 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:04.615646 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:04.615735 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:04.618659 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:36:04.618662 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:36:04.618664 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:36:04.618689 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:36:04.620007 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:04.620463 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:04.620466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:04.620469 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:04.620552 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:04.620844 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:04.620847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:04.620849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:04.621084 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:36:04.621167 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:04.621671 (+ 504us) server_negotiation.cc:299] Negotiation successful
0927 02:36:04.621793 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 02:36:04.622895 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:04.610269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.0.1:39738)
0927 02:36:04.610757 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:04.610809 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:04.611712 (+ 903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:04.612231 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:36:04.612237 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:36:04.612568 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:36:04.613111 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:04.613127 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:04.613928 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:04.613931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:04.615242 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:04.615254 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:04.615814 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:04.615819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:04.616126 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:04.616776 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:36:04.616798 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:36:04.618483 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:36:04.620124 (+ 1641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:04.620129 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:04.620141 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:04.620383 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:04.620619 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:04.620621 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:04.620622 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:04.620742 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:04.621190 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:36:04.621197 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:36:04.621456 (+ 259us) client_negotiation.cc:769] Sending connection context
0927 02:36:04.621617 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 02:36:04.621924 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":137,"threads_started":1}
I20260927 02:36:04.624246 13838 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43729 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 02:36:04.625912 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43891
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33061,127.0.98.61:43729,127.0.98.60:43891
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 02:36:04 dist-test-slave-grlk krb5kdc[13682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 02:36:04 dist-test-slave-grlk krb5kdc[13682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 02:36:04.632301 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:04.622149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43729 (local address 127.0.0.1:53926)
0927 02:36:04.622260 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:04.622274 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:04.622347 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:04.623155 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:36:04.623159 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:36:04.623279 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:36:04.623376 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:04.623410 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:04.624604 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:04.624608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:04.626035 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:04.626045 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:04.626744 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:04.626748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:04.626854 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:04.627264 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:36:04.627277 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:36:04.628810 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:36:04.630700 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:04.630703 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:04.630706 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:04.630952 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:04.631232 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:04.631236 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:04.631238 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:04.631327 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:04.631812 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:36:04.631815 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:36:04.631914 (+ 99us) client_negotiation.cc:769] Sending connection context
0927 02:36:04.631998 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 02:36:04.632142 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 02:36:04.632436 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:04.622409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53926 (local address 127.0.98.61:43729)
0927 02:36:04.622742 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:04.622751 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:04.622769 (+ 18us) server_negotiation.cc:407] Connection header received
0927 02:36:04.622801 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:04.622806 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:04.622988 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:04.623573 (+ 585us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:36:04.623659 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:04.624507 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:04.626161 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:04.626644 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:04.626730 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:04.628972 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:36:04.628982 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:36:04.628997 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:36:04.629023 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:36:04.630585 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:04.631051 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:04.631057 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:04.631059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:04.631148 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:04.631429 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:04.631432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:04.631434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:04.631707 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:36:04.631827 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:04.632130 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 02:36:04.632271 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"spinlock_wait_cycles":3840,"thread_start_us":151,"threads_started":1}
I20260927 02:36:04.633654 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:04.624680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43729 (local address 127.0.0.1:53928)
0927 02:36:04.624757 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:04.624769 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:04.624831 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:04.625285 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:36:04.625287 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:36:04.625412 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:36:04.625503 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:04.625509 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:04.626504 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:04.626508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:04.627431 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:04.627441 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:04.627859 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:04.627863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:04.627954 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:04.628339 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:36:04.628349 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:36:04.629606 (+ 1257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:36:04.632219 (+ 2613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:04.632222 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:04.632224 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:04.632436 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:04.632692 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:04.632694 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:04.632696 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:04.632770 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:04.633216 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:36:04.633218 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:36:04.633296 (+ 78us) client_negotiation.cc:769] Sending connection context
0927 02:36:04.633390 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 02:36:04.633507 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260927 02:36:04.634130 13844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:04.624766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53928 (local address 127.0.98.61:43729)
0927 02:36:04.625045 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:04.625051 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:04.625061 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:36:04.625084 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:04.625086 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:04.625115 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:04.625191 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:36:04.625633 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:04.626401 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:04.627532 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:04.627759 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:04.627847 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:04.630454 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:36:04.630462 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:36:04.630466 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:36:04.630492 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:36:04.632107 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:04.632528 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:04.632530 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:04.632532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:04.632605 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:04.632852 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:04.632855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:04.632857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:04.633095 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:36:04.633183 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:04.633839 (+ 656us) server_negotiation.cc:299] Negotiation successful
0927 02:36:04.633946 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":108,"threads_started":1}
I20260927 02:36:04.634322 13763 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43891 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:36:04.634799 13838 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43891 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:36:04.635010 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:04.634761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43891 (local address 127.0.0.1:59430)
0927 02:36:04.634881 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:04.634937 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 02:36:04.635429 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:04.635173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43891 (local address 127.0.0.1:59440)
0927 02:36:04.635286 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:04.635365 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 02:36:04.635641 13788 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43891: connect: Connection refused (error 111)
I20260927 02:36:04.635869 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:04.635592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43891 (local address 127.0.0.1:59446)
0927 02:36:04.635754 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:04.635805 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260927 02:36:04.636087 13763 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43891: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43891: connect: Connection refused (error 111)
I20260927 02:36:04.636556 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:04.636344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43891 (local address 127.0.0.1:59448)
0927 02:36:04.636447 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:04.636498 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 02:36:04.636695 13838 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43891: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43891: connect: Connection refused (error 111)
I20260927 02:36:04.667270 13763 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43891 } attempt: 1
I20260927 02:36:04.668125 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:04.667742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43891 (local address 127.0.0.1:59464)
0927 02:36:04.667927 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:04.668008 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 02:36:04.668819 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:04.668595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43891 (local address 127.0.0.1:59478)
0927 02:36:04.668704 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:04.668753 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 02:36:04.668954 13763 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43891: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43891: connect: Connection refused (error 111)
I20260927 02:36:04.685958 13838 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43891 } attempt: 1
I20260927 02:36:04.686851 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:04.686496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43891 (local address 127.0.0.1:59484)
0927 02:36:04.686675 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:04.686753 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 02:36:04.687554 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:04.687325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43891 (local address 127.0.0.1:59500)
0927 02:36:04.687438 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:04.687490 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 02:36:04.687680 13838 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43891: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43891: connect: Connection refused (error 111)
W20260927 02:36:04.737965 13845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:36:04.738243 13845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:36:04.738291 13845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:36:04.742332 13845 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:36:04.742386 13845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:36:04.742411 13845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:36:04.742427 13845 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:36:04.742442 13845 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:36:04.744108 13763 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43891 } attempt: 2
I20260927 02:36:04.744968 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:04.744613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43891 (local address 127.0.0.1:59512)
0927 02:36:04.744766 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:04.744835 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 02:36:04.745657 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:04.745449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43891 (local address 127.0.0.1:59522)
0927 02:36:04.745548 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:04.745596 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 02:36:04.745814 13763 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43891: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43891: connect: Connection refused (error 111)
I20260927 02:36:04.748302 13845 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33061,127.0.98.61:43729,127.0.98.60:43891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:36:04.749411 13845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:36:04.750391 13845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:36:04.755838 13838 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43891 } attempt: 2
I20260927 02:36:04.756556 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:04.756299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43891 (local address 127.0.0.1:59538)
0927 02:36:04.756424 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:04.756482 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 02:36:04.757288 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:04.757081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43891 (local address 127.0.0.1:59544)
0927 02:36:04.757176 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:04.757228 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 02:36:04.757413 13838 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43891: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43891: connect: Connection refused (error 111)
W20260927 02:36:04.758846 13854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:36:04.758838 13852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:36:04.758948 13851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:36:04.759181 13845 server_base.cc:1034] running on GCE node
I20260927 02:36:04.759719 13845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:36:04.760809 13845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:36:04.761950 13845 hybrid_clock.cc:648] HybridClock initialized: now 1790476564761923 us; error 39 us; skew 500 ppm
Sep 27 02:36:04 dist-test-slave-grlk krb5kdc[13682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:36:04.764474 13845 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 02:36:04.765329 13845 webserver.cc:460] Webserver started at http://127.0.98.60:41333/ using document root <none> and password file <none>
I20260927 02:36:04.765836 13845 fs_manager.cc:359] Metadata directory not provided
I20260927 02:36:04.765899 13845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:36:04.766074 13845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:36:04.767978 13845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/minicluster-data/master-2/data/instance:
uuid: "728cadb7aa3e4616bb4a7cb9cd6ddfd6"
format_stamp: "Formatted at 2026-09-27 02:36:04 on dist-test-slave-grlk"
server_key: "ed20f4749e45e47759ded6ab24496373"
server_key_iv: "7d770ba35223bf99e65559dfc7627de1"
server_key_version: "encryptionkey@0"
I20260927 02:36:04.768484 13845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/minicluster-data/master-2/wal/instance:
uuid: "728cadb7aa3e4616bb4a7cb9cd6ddfd6"
format_stamp: "Formatted at 2026-09-27 02:36:04 on dist-test-slave-grlk"
server_key: "ed20f4749e45e47759ded6ab24496373"
server_key_iv: "7d770ba35223bf99e65559dfc7627de1"
server_key_version: "encryptionkey@0"
I20260927 02:36:04.771880 13845 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 02:36:04.774144 13861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:04.775277 13845 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:36:04.775381 13845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/minicluster-data/master-2/wal
uuid: "728cadb7aa3e4616bb4a7cb9cd6ddfd6"
format_stamp: "Formatted at 2026-09-27 02:36:04 on dist-test-slave-grlk"
server_key: "ed20f4749e45e47759ded6ab24496373"
server_key_iv: "7d770ba35223bf99e65559dfc7627de1"
server_key_version: "encryptionkey@0"
I20260927 02:36:04.775471 13845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:36:04.788574 13845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:36:04.791389 13845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:36:04.791565 13845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:36:04.798666 13845 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:43891
I20260927 02:36:04.798679 13913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:43891 every 8 connection(s)
I20260927 02:36:04.799582 13845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/minicluster-data/master-2/data/info.pb
I20260927 02:36:04.802209 13914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:36:04.802536 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 13845
I20260927 02:36:04.802672 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476410933508-392-0/minicluster-data/master-2/wal/instance
I20260927 02:36:04.802984 392 external_mini_cluster.cc:1428] Setting key c70ade5eb46fce5d73f4fc810e634959
I20260927 02:36:04.806823 13914 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33061 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:36:04 dist-test-slave-grlk krb5kdc[13682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 02:36:04 dist-test-slave-grlk krb5kdc[13682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:36:04.816875 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:04.804195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59560 (local address 127.0.98.60:43891)
0927 02:36:04.804599 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:04.804614 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:04.804650 (+ 36us) server_negotiation.cc:407] Connection header received
0927 02:36:04.805366 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:04.805384 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:04.805696 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:04.805970 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:36:04.806771 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:04.807778 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:04.808759 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:04.809395 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:04.809518 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:04.811991 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:36:04.811996 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:36:04.812004 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:36:04.812026 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:36:04.814190 (+ 2164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:04.814587 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:04.814593 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:04.814598 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:04.814727 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:04.815033 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:04.815036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:04.815038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:04.815497 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:36:04.815645 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:04.815930 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 02:36:04.816182 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":126,"threads_started":1}
I20260927 02:36:04.816949 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:04.808446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39752 (local address 127.0.98.62:33061)
0927 02:36:04.808590 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:04.808595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:04.808857 (+ 262us) server_negotiation.cc:407] Connection header received
0927 02:36:04.809127 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:04.809130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:04.809176 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:04.809239 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:36:04.809966 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:04.810700 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:04.811564 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:04.811788 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:04.811872 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:04.813882 (+ 2010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:36:04.813886 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:36:04.813889 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:36:04.813916 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:36:04.815325 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:04.815763 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:04.815766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:04.815767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:04.815849 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:04.816129 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:04.816133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:04.816134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:04.816299 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:36:04.816378 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:04.816685 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 02:36:04.816780 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 02:36:04.816962 13918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:04.808371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33061 (local address 127.0.0.1:39752)
0927 02:36:04.808754 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:04.808791 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:04.809005 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:04.809293 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:36:04.809299 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:36:04.809694 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:36:04.809851 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:04.809859 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:04.810827 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:04.810830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:04.811455 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:04.811463 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:04.811940 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:04.811944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:04.812034 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:04.812451 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:36:04.812461 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:36:04.813745 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:36:04.815418 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:04.815426 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:04.815429 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:04.815671 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:04.815949 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:04.815952 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:04.815954 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:04.816040 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:04.816392 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:36:04.816398 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:36:04.816491 (+ 93us) client_negotiation.cc:769] Sending connection context
0927 02:36:04.816599 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 02:36:04.816767 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":146,"threads_started":1}
I20260927 02:36:04.817991 13914 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43729 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:36:04 dist-test-slave-grlk krb5kdc[13682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 02:36:04 dist-test-slave-grlk krb5kdc[13682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:36:04.825794 13918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:04.818401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43729 (local address 127.0.0.1:53936)
0927 02:36:04.818479 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:04.818492 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:04.818553 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:04.818808 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:36:04.818811 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:36:04.818966 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:36:04.819063 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:04.819070 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:04.820405 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:04.820407 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:04.820891 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:04.820897 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:04.821370 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:04.821373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:04.821460 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:04.821809 (+ 349us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:36:04.821818 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:36:04.823030 (+ 1212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:36:04.824512 (+ 1482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:04.824515 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:04.824517 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:04.824684 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:04.824925 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:04.824944 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:04.824946 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:04.825021 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:04.825388 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:36:04.825390 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:36:04.825466 (+ 76us) client_negotiation.cc:769] Sending connection context
0927 02:36:04.825569 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 02:36:04.825684 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260927 02:36:04.826012 13844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:04.818509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53936 (local address 127.0.98.61:43729)
0927 02:36:04.818632 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:04.818636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:04.818649 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:36:04.818685 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:04.818687 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:04.818722 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:04.818783 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:36:04.819177 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:04.819752 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:04.821021 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:04.821270 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:04.821357 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:04.823150 (+ 1793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:36:04.823154 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:36:04.823157 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:36:04.823184 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:36:04.824418 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:04.824761 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:04.824765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:04.824766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:04.824840 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:04.825108 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:04.825112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:04.825114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:04.825288 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:36:04.825384 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:04.825621 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 02:36:04.825738 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 02:36:04.826642 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:04.819040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39768 (local address 127.0.98.62:33061)
0927 02:36:04.819138 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:04.819142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:04.819154 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:36:04.819225 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:04.819228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:04.819264 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:04.819334 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:36:04.819842 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:04.820293 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:04.821065 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:04.821287 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:04.821368 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:04.823649 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:36:04.823652 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:36:04.823653 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:36:04.823673 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:36:04.824991 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:04.825333 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:04.825337 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:04.825338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:04.825414 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:04.825647 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:04.825650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:04.825651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:04.825839 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:36:04.826032 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:04.826388 (+ 356us) server_negotiation.cc:299] Negotiation successful
0927 02:36:04.826495 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 02:36:04.826648 13914 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43891 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:36:04 dist-test-slave-grlk krb5kdc[13682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 02:36:04 dist-test-slave-grlk krb5kdc[13682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 02:36:04.834901 13918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:04.827052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43891 (local address 127.0.0.1:59568)
0927 02:36:04.827129 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:04.827138 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:04.827201 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:04.827432 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:36:04.827435 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:36:04.827575 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:36:04.827654 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:04.827661 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:04.828905 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:04.828908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:04.829689 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:04.829699 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:04.830136 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:04.830138 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:04.830193 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:04.830561 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:36:04.830589 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:36:04.831878 (+ 1289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:36:04.833522 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:04.833525 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:04.833527 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:04.833737 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:04.833956 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:04.833958 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:04.833960 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:04.834038 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:04.834405 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:36:04.834408 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:36:04.834524 (+ 116us) client_negotiation.cc:769] Sending connection context
0927 02:36:04.834637 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 02:36:04.834731 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260927 02:36:04.834913 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:04.827184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59568 (local address 127.0.98.60:43891)
0927 02:36:04.827267 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:04.827271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:04.827283 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:36:04.827309 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:04.827311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:04.827340 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:04.827397 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:36:04.827759 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:04.828804 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:04.829793 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:04.830041 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:04.830127 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:04.831993 (+ 1866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:36:04.831996 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:36:04.831998 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:36:04.832017 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:36:04.833424 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:04.833823 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:04.833825 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:04.833826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:04.833888 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:04.834125 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:04.834128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:04.834130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:04.834320 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:36:04.834416 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:04.834690 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 02:36:04.834795 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 02:36:04.835693 13844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:04.827681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53942 (local address 127.0.98.61:43729)
0927 02:36:04.827779 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:04.827782 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:04.827794 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:36:04.827855 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:04.827857 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:04.827893 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:04.827970 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:36:04.828424 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:04.829090 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:04.829986 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:04.830267 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:04.830345 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:04.832533 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:36:04.832537 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:36:04.832539 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:36:04.832570 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:36:04.834030 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:04.834432 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:04.834436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:04.834437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:04.834515 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:04.834786 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:04.834789 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:04.834791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:04.834982 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:36:04.835174 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:04.835462 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 02:36:04.835567 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 02:36:04.838006 13914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 728cadb7aa3e4616bb4a7cb9cd6ddfd6: Bootstrap starting.
I20260927 02:36:04.838554 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:36:04.840209 13914 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 728cadb7aa3e4616bb4a7cb9cd6ddfd6: Neither blocks nor log segments found. Creating new log.
I20260927 02:36:04.841151 13914 log.cc:824] T 00000000000000000000000000000000 P 728cadb7aa3e4616bb4a7cb9cd6ddfd6: Log is configured to *not* fsync() on all Append() calls
I20260927 02:36:04.843209 13914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 728cadb7aa3e4616bb4a7cb9cd6ddfd6: No bootstrap required, opened a new log
I20260927 02:36:04.845088 13763 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43891 } attempt: 3
I20260927 02:36:04.847178 13914 raft_consensus.cc:348] T 00000000000000000000000000000000 P 728cadb7aa3e4616bb4a7cb9cd6ddfd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde097f4a50045cabbc75590872a17f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33061 } } peers { permanent_uuid: "0bd26fd6f4bb454abb0a1b8da25bd3ab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43729 } } peers { permanent_uuid: "728cadb7aa3e4616bb4a7cb9cd6ddfd6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43891 } }
I20260927 02:36:04.847419 13914 raft_consensus.cc:374] T 00000000000000000000000000000000 P 728cadb7aa3e4616bb4a7cb9cd6ddfd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:36:04.847474 13914 raft_consensus.cc:729] T 00000000000000000000000000000000 P 728cadb7aa3e4616bb4a7cb9cd6ddfd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 728cadb7aa3e4616bb4a7cb9cd6ddfd6, State: Initialized, Role: FOLLOWER
I20260927 02:36:04.848117 13914 consensus_queue.cc:260] T 00000000000000000000000000000000 P 728cadb7aa3e4616bb4a7cb9cd6ddfd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde097f4a50045cabbc75590872a17f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33061 } } peers { permanent_uuid: "0bd26fd6f4bb454abb0a1b8da25bd3ab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43729 } } peers { permanent_uuid: "728cadb7aa3e4616bb4a7cb9cd6ddfd6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43891 } }
I20260927 02:36:04.848543 13838 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43891 } attempt: 3
I20260927 02:36:04.849359 13928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 728cadb7aa3e4616bb4a7cb9cd6ddfd6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde097f4a50045cabbc75590872a17f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33061 } } peers { permanent_uuid: "0bd26fd6f4bb454abb0a1b8da25bd3ab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43729 } } peers { permanent_uuid: "728cadb7aa3e4616bb4a7cb9cd6ddfd6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43891 } } }
I20260927 02:36:04.849462 13914 sys_catalog.cc:564] T 00000000000000000000000000000000 P 728cadb7aa3e4616bb4a7cb9cd6ddfd6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:36:04.849588 13928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 728cadb7aa3e4616bb4a7cb9cd6ddfd6 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:36:04.851965 13844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:04.840715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53946 (local address 127.0.98.61:43729)
0927 02:36:04.840808 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:04.840812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:04.841737 (+ 925us) server_negotiation.cc:407] Connection header received
0927 02:36:04.841893 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:04.841896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:04.841933 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:04.841975 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:36:04.842761 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:04.843469 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:04.844537 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:04.846299 (+ 1762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:04.846369 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:04.847970 (+ 1601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:36:04.847973 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:36:04.847976 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:36:04.847998 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:36:04.849764 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:04.850724 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:04.850728 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:04.850730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:04.850808 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:04.851104 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:04.851108 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:04.851109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:04.851302 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:36:04.851388 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:04.851725 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 02:36:04.851825 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 02:36:04.853303 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:04.840424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59576 (local address 127.0.98.60:43891)
0927 02:36:04.840521 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:04.840524 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:04.841149 (+ 625us) server_negotiation.cc:407] Connection header received
0927 02:36:04.841230 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:04.841233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:04.841270 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:04.841331 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:36:04.841981 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:04.842640 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:04.843726 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:04.844018 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:04.844097 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:04.849572 (+ 5475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:36:04.849576 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:36:04.849578 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:36:04.849602 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:36:04.851294 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:04.851737 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:04.851741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:04.851743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:04.851825 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:04.852404 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:04.852408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:04.852410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:04.852618 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:36:04.852708 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:04.853077 (+ 369us) server_negotiation.cc:299] Negotiation successful
0927 02:36:04.853180 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 02:36:04.853736 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:04.840710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39774 (local address 127.0.98.62:33061)
0927 02:36:04.840808 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:04.840812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:04.840824 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:36:04.840858 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:04.840861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:04.840900 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:04.841005 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:36:04.841811 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:04.842580 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:04.843739 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:04.844066 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:04.844133 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:04.845730 (+ 1597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:36:04.845734 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:36:04.845736 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:36:04.845759 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:36:04.849104 (+ 3345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:04.851136 (+ 2032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:04.851139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:04.851141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:04.851237 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:04.851525 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:04.851528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:04.851530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:04.851754 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:36:04.851840 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:04.853489 (+ 1649us) server_negotiation.cc:299] Negotiation successful
0927 02:36:04.853598 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260927 02:36:04.855916 13940 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 728cadb7aa3e4616bb4a7cb9cd6ddfd6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:36:04.856081 13940 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:36:04.853123 13845 master_runner.cc:186] Error getting master registration for 127.0.98.62:33061: OK, instance_id { permanent_uuid: "fde097f4a50045cabbc75590872a17f3" instance_seqno: 1790476564381733 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 02:36:04.856297 13845 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 27 02:36:04 dist-test-slave-grlk krb5kdc[13682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 02:36:04 dist-test-slave-grlk krb5kdc[13682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 02:36:04.861650 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:04.848836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43891 (local address 127.0.0.1:59594)
0927 02:36:04.849245 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:04.849258 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:04.849322 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:04.850623 (+ 1301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:36:04.850626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:36:04.850843 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:36:04.851019 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:04.851025 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:04.854167 (+ 3142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:04.854171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:04.854940 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:04.854950 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:04.855672 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:04.855676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:04.855749 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:04.856173 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:36:04.856187 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:36:04.857574 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:36:04.860149 (+ 2575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:04.860152 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:04.860154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:04.860359 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:04.860616 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:04.860619 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:04.860620 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:04.860690 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:04.861192 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:36:04.861195 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:36:04.861284 (+ 89us) client_negotiation.cc:769] Sending connection context
0927 02:36:04.861367 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 02:36:04.861486 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342}
I20260927 02:36:04.861680 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:04.849924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59594 (local address 127.0.98.60:43891)
0927 02:36:04.850163 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:04.850172 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:04.850183 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:36:04.850212 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:04.850215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:04.850254 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:04.852365 (+ 2111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:36:04.852475 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:04.853248 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:04.855085 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:04.855510 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:04.856724 (+ 1214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:04.857691 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:36:04.857694 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:36:04.857697 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:36:04.857720 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:36:04.860033 (+ 2313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:04.860441 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:04.860445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:04.860446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:04.860527 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:04.860776 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:04.860779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:04.860781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:04.861112 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:36:04.861204 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:04.861437 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 02:36:04.861533 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":80,"threads_started":1}
I20260927 02:36:04.863420 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:04.845849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43891 (local address 127.0.0.1:59578)
0927 02:36:04.846125 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:04.846137 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:04.846203 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:04.855480 (+ 9277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:36:04.855483 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:36:04.855643 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:36:04.855731 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:04.855738 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:04.856760 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:04.856763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:04.857547 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:04.857559 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:04.858110 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:04.858113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:04.858200 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:04.858569 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:36:04.858582 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:36:04.859898 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:36:04.861971 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:04.861974 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:04.861976 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:04.862216 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:04.862491 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:04.862494 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:04.862496 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:04.862587 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:04.862964 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:36:04.862967 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:36:04.863062 (+ 95us) client_negotiation.cc:769] Sending connection context
0927 02:36:04.863159 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 02:36:04.863275 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260927 02:36:04.863643 13927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:04.845671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59578 (local address 127.0.98.60:43891)
0927 02:36:04.849967 (+ 4296us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:04.849976 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:04.849987 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:36:04.850332 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:04.850335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:04.850378 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:04.855833 (+ 5455us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:36:04.855911 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:04.856646 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:04.857688 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:04.857997 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:04.858102 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:04.860023 (+ 1921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:36:04.860027 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:36:04.860030 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:36:04.860114 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:36:04.861849 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:04.862334 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:04.862337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:04.862339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:04.862415 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:04.862675 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:04.862678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:04.862680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:04.862866 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:36:04.862957 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:04.863218 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 02:36:04.863319 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4208,"thread_start_us":194,"threads_started":1}
I20260927 02:36:04.864774 13838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bd26fd6f4bb454abb0a1b8da25bd3ab: Bootstrap starting.
I20260927 02:36:04.866235 13763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fde097f4a50045cabbc75590872a17f3: Bootstrap starting.
I20260927 02:36:04.867410 13838 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0bd26fd6f4bb454abb0a1b8da25bd3ab: Neither blocks nor log segments found. Creating new log.
I20260927 02:36:04.868233 13838 log.cc:824] T 00000000000000000000000000000000 P 0bd26fd6f4bb454abb0a1b8da25bd3ab: Log is configured to *not* fsync() on all Append() calls
I20260927 02:36:04.868518 13763 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fde097f4a50045cabbc75590872a17f3: Neither blocks nor log segments found. Creating new log.
I20260927 02:36:04.869231 13763 log.cc:824] T 00000000000000000000000000000000 P fde097f4a50045cabbc75590872a17f3: Log is configured to *not* fsync() on all Append() calls
I20260927 02:36:04.870239 13838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bd26fd6f4bb454abb0a1b8da25bd3ab: No bootstrap required, opened a new log
I20260927 02:36:04.870996 13763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fde097f4a50045cabbc75590872a17f3: No bootstrap required, opened a new log
I20260927 02:36:04.874008 13838 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0bd26fd6f4bb454abb0a1b8da25bd3ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde097f4a50045cabbc75590872a17f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33061 } } peers { permanent_uuid: "0bd26fd6f4bb454abb0a1b8da25bd3ab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43729 } } peers { permanent_uuid: "728cadb7aa3e4616bb4a7cb9cd6ddfd6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43891 } }
I20260927 02:36:04.874228 13838 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0bd26fd6f4bb454abb0a1b8da25bd3ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:36:04.874315 13838 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0bd26fd6f4bb454abb0a1b8da25bd3ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bd26fd6f4bb454abb0a1b8da25bd3ab, State: Initialized, Role: FOLLOWER
I20260927 02:36:04.874686 13763 raft_consensus.cc:348] T 00000000000000000000000000000000 P fde097f4a50045cabbc75590872a17f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde097f4a50045cabbc75590872a17f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33061 } } peers { permanent_uuid: "0bd26fd6f4bb454abb0a1b8da25bd3ab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43729 } } peers { permanent_uuid: "728cadb7aa3e4616bb4a7cb9cd6ddfd6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43891 } }
I20260927 02:36:04.874752 13838 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bd26fd6f4bb454abb0a1b8da25bd3ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde097f4a50045cabbc75590872a17f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33061 } } peers { permanent_uuid: "0bd26fd6f4bb454abb0a1b8da25bd3ab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43729 } } peers { permanent_uuid: "728cadb7aa3e4616bb4a7cb9cd6ddfd6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43891 } }
I20260927 02:36:04.874871 13763 raft_consensus.cc:374] T 00000000000000000000000000000000 P fde097f4a50045cabbc75590872a17f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:36:04.874962 13763 raft_consensus.cc:729] T 00000000000000000000000000000000 P fde097f4a50045cabbc75590872a17f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde097f4a50045cabbc75590872a17f3, State: Initialized, Role: FOLLOWER
I20260927 02:36:04.875432 13944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bd26fd6f4bb454abb0a1b8da25bd3ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde097f4a50045cabbc75590872a17f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33061 } } peers { permanent_uuid: "0bd26fd6f4bb454abb0a1b8da25bd3ab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43729 } } peers { permanent_uuid: "728cadb7aa3e4616bb4a7cb9cd6ddfd6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43891 } } }
I20260927 02:36:04.875432 13763 consensus_queue.cc:260] T 00000000000000000000000000000000 P fde097f4a50045cabbc75590872a17f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde097f4a50045cabbc75590872a17f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33061 } } peers { permanent_uuid: "0bd26fd6f4bb454abb0a1b8da25bd3ab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43729 } } peers { permanent_uuid: "728cadb7aa3e4616bb4a7cb9cd6ddfd6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43891 } }
I20260927 02:36:04.875624 13944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bd26fd6f4bb454abb0a1b8da25bd3ab [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:36:04.875958 13838 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0bd26fd6f4bb454abb0a1b8da25bd3ab [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:36:04.876536 13763 sys_catalog.cc:564] T 00000000000000000000000000000000 P fde097f4a50045cabbc75590872a17f3 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 02:36:04.878307 13716 tablet.cc:2369] T 00000000000000000000000000000000 P fde097f4a50045cabbc75590872a17f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:36:04.878939 13945 sys_catalog.cc:455] T 00000000000000000000000000000000 P fde097f4a50045cabbc75590872a17f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde097f4a50045cabbc75590872a17f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33061 } } peers { permanent_uuid: "0bd26fd6f4bb454abb0a1b8da25bd3ab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43729 } } peers { permanent_uuid: "728cadb7aa3e4616bb4a7cb9cd6ddfd6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43891 } } }
I20260927 02:36:04.881165 13945 sys_catalog.cc:458] T 00000000000000000000000000000000 P fde097f4a50045cabbc75590872a17f3 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 02:36:04.883419 13967 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fde097f4a50045cabbc75590872a17f3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:36:04.883481 13967 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260927 02:36:04.884948 13961 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0bd26fd6f4bb454abb0a1b8da25bd3ab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:36:04.885020 13961 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:36:04.885694 13695 master_runner.cc:263] No leader master found from master fde097f4a50045cabbc75590872a17f3
I20260927 02:36:04.885751 13695 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 02:36:04.886463 13770 master_runner.cc:263] No leader master found from master 0bd26fd6f4bb454abb0a1b8da25bd3ab
I20260927 02:36:04.886523 13770 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 02:36:04.916123 13944 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bd26fd6f4bb454abb0a1b8da25bd3ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:36:04.916390 13944 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bd26fd6f4bb454abb0a1b8da25bd3ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde097f4a50045cabbc75590872a17f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33061 } } peers { permanent_uuid: "0bd26fd6f4bb454abb0a1b8da25bd3ab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43729 } } peers { permanent_uuid: "728cadb7aa3e4616bb4a7cb9cd6ddfd6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43891 } }
I20260927 02:36:04.917938 13737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0bd26fd6f4bb454abb0a1b8da25bd3ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde097f4a50045cabbc75590872a17f3" is_pre_election: true
I20260927 02:36:04.918259 13737 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fde097f4a50045cabbc75590872a17f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bd26fd6f4bb454abb0a1b8da25bd3ab in term 0.
I20260927 02:36:04.918774 13786 leader_election.cc:302] T 00000000000000000000000000000000 P 0bd26fd6f4bb454abb0a1b8da25bd3ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bd26fd6f4bb454abb0a1b8da25bd3ab, fde097f4a50045cabbc75590872a17f3; no voters:
I20260927 02:36:04.921759 13888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0bd26fd6f4bb454abb0a1b8da25bd3ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "728cadb7aa3e4616bb4a7cb9cd6ddfd6" is_pre_election: true
I20260927 02:36:04.922051 13888 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 728cadb7aa3e4616bb4a7cb9cd6ddfd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bd26fd6f4bb454abb0a1b8da25bd3ab in term 0.
I20260927 02:36:04.922542 13944 leader_election.cc:288] T 00000000000000000000000000000000 P 0bd26fd6f4bb454abb0a1b8da25bd3ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fde097f4a50045cabbc75590872a17f3 (127.0.98.62:33061), 728cadb7aa3e4616bb4a7cb9cd6ddfd6 (127.0.98.60:43891)
I20260927 02:36:04.922869 13944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0bd26fd6f4bb454abb0a1b8da25bd3ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:36:04.923009 13944 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bd26fd6f4bb454abb0a1b8da25bd3ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:36:04.923118 13944 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bd26fd6f4bb454abb0a1b8da25bd3ab [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:36:04.924417 13944 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bd26fd6f4bb454abb0a1b8da25bd3ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde097f4a50045cabbc75590872a17f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33061 } } peers { permanent_uuid: "0bd26fd6f4bb454abb0a1b8da25bd3ab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43729 } } peers { permanent_uuid: "728cadb7aa3e4616bb4a7cb9cd6ddfd6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43891 } }
I20260927 02:36:04.925061 13737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0bd26fd6f4bb454abb0a1b8da25bd3ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde097f4a50045cabbc75590872a17f3"
I20260927 02:36:04.925184 13737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fde097f4a50045cabbc75590872a17f3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:36:04.926363 13737 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fde097f4a50045cabbc75590872a17f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bd26fd6f4bb454abb0a1b8da25bd3ab in term 1.
I20260927 02:36:04.926712 13786 leader_election.cc:302] T 00000000000000000000000000000000 P 0bd26fd6f4bb454abb0a1b8da25bd3ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bd26fd6f4bb454abb0a1b8da25bd3ab, fde097f4a50045cabbc75590872a17f3; no voters:
I20260927 02:36:04.927189 13888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0bd26fd6f4bb454abb0a1b8da25bd3ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "728cadb7aa3e4616bb4a7cb9cd6ddfd6"
I20260927 02:36:04.927294 13888 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 728cadb7aa3e4616bb4a7cb9cd6ddfd6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:36:04.928422 13888 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 728cadb7aa3e4616bb4a7cb9cd6ddfd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bd26fd6f4bb454abb0a1b8da25bd3ab in term 1.
I20260927 02:36:04.928756 13944 leader_election.cc:288] T 00000000000000000000000000000000 P 0bd26fd6f4bb454abb0a1b8da25bd3ab [CANDIDATE]: Term 1 election: Requested vote from peers fde097f4a50045cabbc75590872a17f3 (127.0.98.62:33061), 728cadb7aa3e4616bb4a7cb9cd6ddfd6 (127.0.98.60:43891)
I20260927 02:36:04.928923 13944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0bd26fd6f4bb454abb0a1b8da25bd3ab [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:36:04.929064 13944 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0bd26fd6f4bb454abb0a1b8da25bd3ab [term 1 LEADER]: Becoming Leader. State: Replica: 0bd26fd6f4bb454abb0a1b8da25bd3ab, State: Running, Role: LEADER
I20260927 02:36:04.929409 13944 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0bd26fd6f4bb454abb0a1b8da25bd3ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde097f4a50045cabbc75590872a17f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33061 } } peers { permanent_uuid: "0bd26fd6f4bb454abb0a1b8da25bd3ab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43729 } } peers { permanent_uuid: "728cadb7aa3e4616bb4a7cb9cd6ddfd6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43891 } }
I20260927 02:36:04.931111 13944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bd26fd6f4bb454abb0a1b8da25bd3ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bd26fd6f4bb454abb0a1b8da25bd3ab. Latest consensus state: current_term: 1 leader_uuid: "0bd26fd6f4bb454abb0a1b8da25bd3ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde097f4a50045cabbc75590872a17f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33061 } } peers { permanent_uuid: "0bd26fd6f4bb454abb0a1b8da25bd3ab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43729 } } peers { permanent_uuid: "728cadb7aa3e4616bb4a7cb9cd6ddfd6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43891 } } }
I20260927 02:36:04.931396 13944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bd26fd6f4bb454abb0a1b8da25bd3ab [sys.catalog]: This master's current role is: LEADER
I20260927 02:36:04.935104 13972 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:36:04.936053 13972 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:36:04.941304 13888 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 728cadb7aa3e4616bb4a7cb9cd6ddfd6 [term 1 FOLLOWER]: Refusing update from remote peer 0bd26fd6f4bb454abb0a1b8da25bd3ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:36:04.941983 13944 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0bd26fd6f4bb454abb0a1b8da25bd3ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "728cadb7aa3e4616bb4a7cb9cd6ddfd6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:36:04.949491 13928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 728cadb7aa3e4616bb4a7cb9cd6ddfd6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bd26fd6f4bb454abb0a1b8da25bd3ab. Latest consensus state: current_term: 1 leader_uuid: "0bd26fd6f4bb454abb0a1b8da25bd3ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde097f4a50045cabbc75590872a17f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33061 } } peers { permanent_uuid: "0bd26fd6f4bb454abb0a1b8da25bd3ab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43729 } } peers { permanent_uuid: "728cadb7aa3e4616bb4a7cb9cd6ddfd6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43891 } } }
I20260927 02:36:04.949615 13928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 728cadb7aa3e4616bb4a7cb9cd6ddfd6 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:36:04.950950 13737 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fde097f4a50045cabbc75590872a17f3 [term 1 FOLLOWER]: Refusing update from remote peer 0bd26fd6f4bb454abb0a1b8da25bd3ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:36:04.951551 13944 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0bd26fd6f4bb454abb0a1b8da25bd3ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde097f4a50045cabbc75590872a17f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:36:04.955693 13945 sys_catalog.cc:455] T 00000000000000000000000000000000 P fde097f4a50045cabbc75590872a17f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bd26fd6f4bb454abb0a1b8da25bd3ab. Latest consensus state: current_term: 1 leader_uuid: "0bd26fd6f4bb454abb0a1b8da25bd3ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde097f4a50045cabbc75590872a17f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33061 } } peers { permanent_uuid: "0bd26fd6f4bb454abb0a1b8da25bd3ab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43729 } } peers { permanent_uuid: "728cadb7aa3e4616bb4a7cb9cd6ddfd6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43891 } } }
I20260927 02:36:04.955816 13945 sys_catalog.cc:458] T 00000000000000000000000000000000 P fde097f4a50045cabbc75590872a17f3 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:36:04.959408 13944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bd26fd6f4bb454abb0a1b8da25bd3ab [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0bd26fd6f4bb454abb0a1b8da25bd3ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde097f4a50045cabbc75590872a17f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33061 } } peers { permanent_uuid: "0bd26fd6f4bb454abb0a1b8da25bd3ab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43729 } } peers { permanent_uuid: "728cadb7aa3e4616bb4a7cb9cd6ddfd6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43891 } } }
I20260927 02:36:04.959543 13944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bd26fd6f4bb454abb0a1b8da25bd3ab [sys.catalog]: This master's current role is: LEADER
I20260927 02:36:04.960716 13945 sys_catalog.cc:455] T 00000000000000000000000000000000 P fde097f4a50045cabbc75590872a17f3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0bd26fd6f4bb454abb0a1b8da25bd3ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde097f4a50045cabbc75590872a17f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33061 } } peers { permanent_uuid: "0bd26fd6f4bb454abb0a1b8da25bd3ab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43729 } } peers { permanent_uuid: "728cadb7aa3e4616bb4a7cb9cd6ddfd6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43891 } } }
I20260927 02:36:04.960824 13945 sys_catalog.cc:458] T 00000000000000000000000000000000 P fde097f4a50045cabbc75590872a17f3 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:36:04.961264 13973 mvcc.cc:205] Tried to move back new op lower bound from 7333792009992433664 to 7333792009953878016. Current Snapshot: MvccSnapshot[applied={T|T < 7333792009992433664}]
I20260927 02:36:04.966212 13976 mvcc.cc:205] Tried to move back new op lower bound from 7333792009992433664 to 7333792009953878016. Current Snapshot: MvccSnapshot[applied={T|T < 7333792009992433664}]
I20260927 02:36:04.967195 13970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bd26fd6f4bb454abb0a1b8da25bd3ab [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0bd26fd6f4bb454abb0a1b8da25bd3ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde097f4a50045cabbc75590872a17f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33061 } } peers { permanent_uuid: "0bd26fd6f4bb454abb0a1b8da25bd3ab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43729 } } peers { permanent_uuid: "728cadb7aa3e4616bb4a7cb9cd6ddfd6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43891 } } }
I20260927 02:36:04.967309 13970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bd26fd6f4bb454abb0a1b8da25bd3ab [sys.catalog]: This master's current role is: LEADER
I20260927 02:36:04.968278 13928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 728cadb7aa3e4616bb4a7cb9cd6ddfd6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0bd26fd6f4bb454abb0a1b8da25bd3ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde097f4a50045cabbc75590872a17f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33061 } } peers { permanent_uuid: "0bd26fd6f4bb454abb0a1b8da25bd3ab" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43729 } } peers { permanent_uuid: "728cadb7aa3e4616bb4a7cb9cd6ddfd6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43891 } } }
I20260927 02:36:04.968396 13928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 728cadb7aa3e4616bb4a7cb9cd6ddfd6 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:36:04.970811 13974 mvcc.cc:205] Tried to move back new op lower bound from 7333792009992433664 to 7333792009953878016. Current Snapshot: MvccSnapshot[applied={T|T < 7333792009992433664}]
I20260927 02:36:04.973054 13972 catalog_manager.cc:1347] Generated new cluster ID: 9b3332c3597a4b96aa9036b260f5972e
I20260927 02:36:04.973112 13972 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:36:04.991791 13972 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:36:04.995611 13972 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:36:05.017258 13972 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0bd26fd6f4bb454abb0a1b8da25bd3ab: Generated new TSK 0
I20260927 02:36:05.018108 13972 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:36:05.857686 13940 catalog_manager.cc:1259] Loaded cluster ID: 9b3332c3597a4b96aa9036b260f5972e
I20260927 02:36:05.857762 13940 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 728cadb7aa3e4616bb4a7cb9cd6ddfd6: loading cluster ID for follower catalog manager: success
I20260927 02:36:05.859516 13940 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 728cadb7aa3e4616bb4a7cb9cd6ddfd6: acquiring CA information for follower catalog manager: success
I20260927 02:36:05.860702 13940 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 728cadb7aa3e4616bb4a7cb9cd6ddfd6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 02:36:05.884682 13967 catalog_manager.cc:1259] Loaded cluster ID: 9b3332c3597a4b96aa9036b260f5972e
I20260927 02:36:05.884759 13967 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fde097f4a50045cabbc75590872a17f3: loading cluster ID for follower catalog manager: success
I20260927 02:36:05.886713 13967 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fde097f4a50045cabbc75590872a17f3: acquiring CA information for follower catalog manager: success
I20260927 02:36:05.887820 13967 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fde097f4a50045cabbc75590872a17f3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 02:36:06.084074 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 13695
I20260927 02:36:06.089381 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 13770
I20260927 02:36:06.094483 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 13845
2026-09-27T02:36:06Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2958 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476410933508-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:36:06 dist-test-slave-grlk krb5kdc[13989](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:36:06 dist-test-slave-grlk krb5kdc[13989](info): set up 2 sockets
Sep 27 02:36:06 dist-test-slave-grlk krb5kdc[13989](info): commencing operation
krb5kdc: starting...
W20260927 02:36:07.048900 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476410933508-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476410933508-392-0/krb5kdc/test-user.keytab.
Sep 27 02:36:07 dist-test-slave-grlk krb5kdc[13989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:36:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:36:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:36:07.160147 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35761
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35761
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:36:07.264252 14002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:36:07.264534 14002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:36:07.264595 14002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:36:07.268519 14002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:36:07.268577 14002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:36:07.268604 14002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:36:07.268626 14002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:36:07.268654 14002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:36:07.273746 14002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:36:07.274704 14002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:36:07.275647 14002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:36:07.283784 14008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:36:07.283922 14002 server_base.cc:1034] running on GCE node
W20260927 02:36:07.283733 14007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:36:07.283716 14010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:36:07.284497 14002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:36:07.285553 14002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:36:07.286695 14002 hybrid_clock.cc:648] HybridClock initialized: now 1790476567286680 us; error 29 us; skew 500 ppm
Sep 27 02:36:07 dist-test-slave-grlk krb5kdc[13989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:36:07.289232 14002 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:36:07.290038 14002 webserver.cc:460] Webserver started at http://127.0.98.62:36969/ using document root <none> and password file <none>
I20260927 02:36:07.290534 14002 fs_manager.cc:359] Metadata directory not provided
I20260927 02:36:07.290591 14002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:36:07.290772 14002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:36:07.292682 14002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "e36ecd1246e24fa68c525f070b86b0b0"
format_stamp: "Formatted at 2026-09-27 02:36:07 on dist-test-slave-grlk"
server_key: "494403548495d72167b4e3d31e0dd6d2"
server_key_iv: "bd6d8d32dcae2166e3f7ae32d07e6674"
server_key_version: "encryptionkey@0"
I20260927 02:36:07.293246 14002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "e36ecd1246e24fa68c525f070b86b0b0"
format_stamp: "Formatted at 2026-09-27 02:36:07 on dist-test-slave-grlk"
server_key: "494403548495d72167b4e3d31e0dd6d2"
server_key_iv: "bd6d8d32dcae2166e3f7ae32d07e6674"
server_key_version: "encryptionkey@0"
I20260927 02:36:07.296681 14002 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260927 02:36:07.298957 14017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:07.300051 14002 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:36:07.300150 14002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "e36ecd1246e24fa68c525f070b86b0b0"
format_stamp: "Formatted at 2026-09-27 02:36:07 on dist-test-slave-grlk"
server_key: "494403548495d72167b4e3d31e0dd6d2"
server_key_iv: "bd6d8d32dcae2166e3f7ae32d07e6674"
server_key_version: "encryptionkey@0"
I20260927 02:36:07.300246 14002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:36:07.318441 14002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:36:07.321422 14002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:36:07.321586 14002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:36:07.328143 14002 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35761
I20260927 02:36:07.328176 14069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35761 every 8 connection(s)
I20260927 02:36:07.329062 14002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:36:07.331604 14070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:36:07.335340 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 14002
I20260927 02:36:07.335461 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:36:07.335748 392 external_mini_cluster.cc:1428] Setting key 636e297eaebffd0b4d9ec9f93427fcf8
I20260927 02:36:07.337164 14070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e36ecd1246e24fa68c525f070b86b0b0: Bootstrap starting.
I20260927 02:36:07.339229 14070 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e36ecd1246e24fa68c525f070b86b0b0: Neither blocks nor log segments found. Creating new log.
I20260927 02:36:07.339911 14070 log.cc:824] T 00000000000000000000000000000000 P e36ecd1246e24fa68c525f070b86b0b0: Log is configured to *not* fsync() on all Append() calls
I20260927 02:36:07.341739 14070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e36ecd1246e24fa68c525f070b86b0b0: No bootstrap required, opened a new log
Sep 27 02:36:07 dist-test-slave-grlk krb5kdc[13989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:36:07.345347 14070 raft_consensus.cc:348] T 00000000000000000000000000000000 P e36ecd1246e24fa68c525f070b86b0b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e36ecd1246e24fa68c525f070b86b0b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35761 } }
I20260927 02:36:07.345557 14070 raft_consensus.cc:374] T 00000000000000000000000000000000 P e36ecd1246e24fa68c525f070b86b0b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:36:07.345602 14070 raft_consensus.cc:729] T 00000000000000000000000000000000 P e36ecd1246e24fa68c525f070b86b0b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e36ecd1246e24fa68c525f070b86b0b0, State: Initialized, Role: FOLLOWER
I20260927 02:36:07.346014 14070 consensus_queue.cc:260] T 00000000000000000000000000000000 P e36ecd1246e24fa68c525f070b86b0b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e36ecd1246e24fa68c525f070b86b0b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35761 } }
I20260927 02:36:07.346143 14070 raft_consensus.cc:388] T 00000000000000000000000000000000 P e36ecd1246e24fa68c525f070b86b0b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:36:07.346201 14070 raft_consensus.cc:482] T 00000000000000000000000000000000 P e36ecd1246e24fa68c525f070b86b0b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:36:07.346290 14070 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e36ecd1246e24fa68c525f070b86b0b0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:36:07.347096 14070 raft_consensus.cc:504] T 00000000000000000000000000000000 P e36ecd1246e24fa68c525f070b86b0b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e36ecd1246e24fa68c525f070b86b0b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35761 } }
I20260927 02:36:07.347397 14070 leader_election.cc:302] T 00000000000000000000000000000000 P e36ecd1246e24fa68c525f070b86b0b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e36ecd1246e24fa68c525f070b86b0b0; no voters:
I20260927 02:36:07.347672 14070 leader_election.cc:288] T 00000000000000000000000000000000 P e36ecd1246e24fa68c525f070b86b0b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:36:07.347776 14075 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e36ecd1246e24fa68c525f070b86b0b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:36:07.347987 14075 raft_consensus.cc:686] T 00000000000000000000000000000000 P e36ecd1246e24fa68c525f070b86b0b0 [term 1 LEADER]: Becoming Leader. State: Replica: e36ecd1246e24fa68c525f070b86b0b0, State: Running, Role: LEADER
I20260927 02:36:07.348312 14075 consensus_queue.cc:237] T 00000000000000000000000000000000 P e36ecd1246e24fa68c525f070b86b0b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e36ecd1246e24fa68c525f070b86b0b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35761 } }
I20260927 02:36:07.348958 14070 sys_catalog.cc:564] T 00000000000000000000000000000000 P e36ecd1246e24fa68c525f070b86b0b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:36:07.349953 14076 sys_catalog.cc:455] T 00000000000000000000000000000000 P e36ecd1246e24fa68c525f070b86b0b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e36ecd1246e24fa68c525f070b86b0b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e36ecd1246e24fa68c525f070b86b0b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35761 } } }
I20260927 02:36:07.350080 14076 sys_catalog.cc:458] T 00000000000000000000000000000000 P e36ecd1246e24fa68c525f070b86b0b0 [sys.catalog]: This master's current role is: LEADER
I20260927 02:36:07.350255 14073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:07.337047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54182 (local address 127.0.98.62:35761)
0927 02:36:07.337440 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:07.337454 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:07.337487 (+ 33us) server_negotiation.cc:407] Connection header received
0927 02:36:07.338101 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:07.338117 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:07.338421 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:07.338710 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:36:07.339480 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:07.340502 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:07.341811 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:07.342311 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:07.342459 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:07.345027 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:36:07.345031 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:36:07.345044 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:36:07.345070 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:36:07.346757 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:07.347152 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:07.347158 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:07.347164 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:07.347287 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:07.347553 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:07.347557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:07.347559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:07.348069 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:36:07.348819 (+ 750us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:07.349002 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 02:36:07.349293 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":118,"threads_started":1}
I20260927 02:36:07.350509 14077 sys_catalog.cc:455] T 00000000000000000000000000000000 P e36ecd1246e24fa68c525f070b86b0b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e36ecd1246e24fa68c525f070b86b0b0. Latest consensus state: current_term: 1 leader_uuid: "e36ecd1246e24fa68c525f070b86b0b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e36ecd1246e24fa68c525f070b86b0b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35761 } } }
I20260927 02:36:07.350600 14077 sys_catalog.cc:458] T 00000000000000000000000000000000 P e36ecd1246e24fa68c525f070b86b0b0 [sys.catalog]: This master's current role is: LEADER
I20260927 02:36:07.350968 14084 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:36:07.353124 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:36:07.354182 14084 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:36:07.359522 14084 catalog_manager.cc:1347] Generated new cluster ID: 78b25838bb13448bb033c8eda69dd46d
I20260927 02:36:07.359586 14084 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:36:07.374449 14084 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:36:07.375351 14084 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:36:07.384034 14084 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e36ecd1246e24fa68c525f070b86b0b0: Generated new TSK 0
I20260927 02:36:07.384625 14084 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:36:07.408432 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 14002
2026-09-27T02:36:07Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1311 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:36:07 dist-test-slave-grlk krb5kdc[14097](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:36:07 dist-test-slave-grlk krb5kdc[14097](info): set up 2 sockets
Sep 27 02:36:07 dist-test-slave-grlk krb5kdc[14097](info): commencing operation
krb5kdc: starting...
W20260927 02:36:08.365434 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/test-user.keytab.
Sep 27 02:36:08 dist-test-slave-grlk krb5kdc[14097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:36:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:36:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:36:08.493389 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36259
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36259
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:36:08.609001 14110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:36:08.609243 14110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:36:08.609284 14110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:36:08.613171 14110 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:36:08.613216 14110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:36:08.613233 14110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:36:08.613251 14110 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:36:08.613279 14110 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:36:08.618412 14110 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:36:08.619379 14110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:36:08.620330 14110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:36:08.628229 14116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:36:08.628297 14118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:36:08.628239 14115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:36:08.628398 14110 server_base.cc:1034] running on GCE node
I20260927 02:36:08.629184 14110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:36:08.630173 14110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:36:08.631325 14110 hybrid_clock.cc:648] HybridClock initialized: now 1790476568631306 us; error 31 us; skew 500 ppm
Sep 27 02:36:08 dist-test-slave-grlk krb5kdc[14097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476568, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:36:08.633869 14110 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:36:08.634714 14110 webserver.cc:460] Webserver started at http://127.0.98.62:42597/ using document root <none> and password file <none>
I20260927 02:36:08.635227 14110 fs_manager.cc:359] Metadata directory not provided
I20260927 02:36:08.635293 14110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:36:08.635474 14110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:36:08.637656 14110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "b53ecc8c654d41c7a4e117f1ef1d345e"
format_stamp: "Formatted at 2026-09-27 02:36:08 on dist-test-slave-grlk"
server_key: "c22e67050e2607299641684c3b867e42"
server_key_iv: "eefafd85429342ad55b0acd8de8f4e9a"
server_key_version: "encryptionkey@0"
I20260927 02:36:08.638185 14110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "b53ecc8c654d41c7a4e117f1ef1d345e"
format_stamp: "Formatted at 2026-09-27 02:36:08 on dist-test-slave-grlk"
server_key: "c22e67050e2607299641684c3b867e42"
server_key_iv: "eefafd85429342ad55b0acd8de8f4e9a"
server_key_version: "encryptionkey@0"
I20260927 02:36:08.641682 14110 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:36:08.644004 14125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:08.645208 14110 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:36:08.645318 14110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "b53ecc8c654d41c7a4e117f1ef1d345e"
format_stamp: "Formatted at 2026-09-27 02:36:08 on dist-test-slave-grlk"
server_key: "c22e67050e2607299641684c3b867e42"
server_key_iv: "eefafd85429342ad55b0acd8de8f4e9a"
server_key_version: "encryptionkey@0"
I20260927 02:36:08.645408 14110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:36:08.659590 14110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:36:08.661993 14110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:36:08.662154 14110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:36:08.668237 14110 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36259
I20260927 02:36:08.668468 14177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36259 every 8 connection(s)
I20260927 02:36:08.669153 14110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:36:08.669883 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 14110
I20260927 02:36:08.670010 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:36:08.670307 392 external_mini_cluster.cc:1428] Setting key e8044d2f240c2d03bc6b426611ac5468
I20260927 02:36:08.672091 14178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:36:08.678252 14178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e: Bootstrap starting.
Sep 27 02:36:08 dist-test-slave-grlk krb5kdc[14097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:36:08.680279 14178 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e: Neither blocks nor log segments found. Creating new log.
I20260927 02:36:08.680965 14178 log.cc:824] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e: Log is configured to *not* fsync() on all Append() calls
I20260927 02:36:08.682694 14178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e: No bootstrap required, opened a new log
I20260927 02:36:08.683105 14181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:08.671505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55126 (local address 127.0.98.62:36259)
0927 02:36:08.671864 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:08.671886 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:08.671927 (+ 41us) server_negotiation.cc:407] Connection header received
0927 02:36:08.672573 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:08.672588 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:08.672888 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:08.673210 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:36:08.674001 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:08.674974 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:08.676041 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:08.676472 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:08.676583 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:08.679096 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:36:08.679100 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:36:08.679113 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:36:08.679137 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:36:08.680594 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:08.680980 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:08.680984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:08.680988 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:08.681098 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:08.681344 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:08.681347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:08.681349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:08.681709 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:36:08.681885 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:08.682170 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 02:36:08.682422 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":99,"threads_started":1}
I20260927 02:36:08.686232 14178 raft_consensus.cc:348] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53ecc8c654d41c7a4e117f1ef1d345e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36259 } }
I20260927 02:36:08.686434 14178 raft_consensus.cc:374] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:36:08.686488 14178 raft_consensus.cc:729] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b53ecc8c654d41c7a4e117f1ef1d345e, State: Initialized, Role: FOLLOWER
I20260927 02:36:08.686856 14178 consensus_queue.cc:260] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53ecc8c654d41c7a4e117f1ef1d345e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36259 } }
I20260927 02:36:08.686972 14178 raft_consensus.cc:388] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:36:08.687031 14178 raft_consensus.cc:482] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:36:08.687120 14178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:36:08.687882 14178 raft_consensus.cc:504] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53ecc8c654d41c7a4e117f1ef1d345e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36259 } }
I20260927 02:36:08.688169 14178 leader_election.cc:302] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b53ecc8c654d41c7a4e117f1ef1d345e; no voters:
I20260927 02:36:08.688421 14178 leader_election.cc:288] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:36:08.688498 14183 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:36:08.688705 14183 raft_consensus.cc:686] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [term 1 LEADER]: Becoming Leader. State: Replica: b53ecc8c654d41c7a4e117f1ef1d345e, State: Running, Role: LEADER
I20260927 02:36:08.689056 14183 consensus_queue.cc:237] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53ecc8c654d41c7a4e117f1ef1d345e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36259 } }
I20260927 02:36:08.689536 14178 sys_catalog.cc:564] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:36:08.690579 14184 sys_catalog.cc:455] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b53ecc8c654d41c7a4e117f1ef1d345e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53ecc8c654d41c7a4e117f1ef1d345e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36259 } } }
I20260927 02:36:08.690614 14185 sys_catalog.cc:455] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b53ecc8c654d41c7a4e117f1ef1d345e. Latest consensus state: current_term: 1 leader_uuid: "b53ecc8c654d41c7a4e117f1ef1d345e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53ecc8c654d41c7a4e117f1ef1d345e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36259 } } }
I20260927 02:36:08.690694 14184 sys_catalog.cc:458] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [sys.catalog]: This master's current role is: LEADER
I20260927 02:36:08.690702 14185 sys_catalog.cc:458] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [sys.catalog]: This master's current role is: LEADER
I20260927 02:36:08.691027 14192 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:36:08.693739 14192 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:36:08.698684 14192 catalog_manager.cc:1347] Generated new cluster ID: f86887c5d3c2418f83ba2792ec8f45bf
I20260927 02:36:08.698748 14192 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:36:08.722990 14192 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:36:08.724254 14192 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:36:08.731931 14192 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e: Generated new TSK 0
I20260927 02:36:08.732712 14192 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:36:08.763571 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36259
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:36:08.874132 14204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:36:08.874356 14204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:36:08.874397 14204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:36:08.878320 14204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:36:08.878366 14204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:36:08.878425 14204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:36:08.883661 14204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:36:08.884634 14204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:36:08.885612 14204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:36:08.894538 14210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:36:08.894652 14204 server_base.cc:1034] running on GCE node
W20260927 02:36:08.894580 14209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:36:08.894573 14212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:36:08.895135 14204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:36:08.895731 14204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:36:08.897004 14204 hybrid_clock.cc:648] HybridClock initialized: now 1790476568896994 us; error 25 us; skew 500 ppm
Sep 27 02:36:08 dist-test-slave-grlk krb5kdc[14097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476568, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:36:08.899555 14204 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:36:08.900388 14204 webserver.cc:460] Webserver started at http://127.0.98.1:43069/ using document root <none> and password file <none>
I20260927 02:36:08.900902 14204 fs_manager.cc:359] Metadata directory not provided
I20260927 02:36:08.900996 14204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:36:08.901189 14204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:36:08.903074 14204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-0/data/instance:
uuid: "d978229ba60143278b1226ab8132033d"
format_stamp: "Formatted at 2026-09-27 02:36:08 on dist-test-slave-grlk"
server_key: "667717698431fe3ce035658e6d4e7f21"
server_key_iv: "96c13820116b71566586b6eeec4bc1f8"
server_key_version: "encryptionkey@0"
I20260927 02:36:08.903571 14204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d978229ba60143278b1226ab8132033d"
format_stamp: "Formatted at 2026-09-27 02:36:08 on dist-test-slave-grlk"
server_key: "667717698431fe3ce035658e6d4e7f21"
server_key_iv: "96c13820116b71566586b6eeec4bc1f8"
server_key_version: "encryptionkey@0"
I20260927 02:36:08.906960 14204 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:36:08.909108 14219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:08.910207 14204 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:36:08.910312 14204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-0/wal
uuid: "d978229ba60143278b1226ab8132033d"
format_stamp: "Formatted at 2026-09-27 02:36:08 on dist-test-slave-grlk"
server_key: "667717698431fe3ce035658e6d4e7f21"
server_key_iv: "96c13820116b71566586b6eeec4bc1f8"
server_key_version: "encryptionkey@0"
I20260927 02:36:08.910394 14204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:36:08.952555 14204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:36:08.955504 14204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:36:08.955681 14204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:36:08.956230 14204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:36:08.957056 14204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:36:08.957106 14204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:08.957160 14204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:36:08.957187 14204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:08.965301 14204 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45151
I20260927 02:36:08.965339 14332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45151 every 8 connection(s)
I20260927 02:36:08.966113 14204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:36:08.969388 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 14204
I20260927 02:36:08.969480 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:36:08.969726 392 external_mini_cluster.cc:1428] Setting key 4c5d3d43ae1bd416ca1f4fa44764550b
Sep 27 02:36:08 dist-test-slave-grlk krb5kdc[14097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476568, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:36:08.978667 14181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:08.967694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38441 (local address 127.0.98.62:36259)
0927 02:36:08.967825 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:08.967829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:08.968603 (+ 774us) server_negotiation.cc:407] Connection header received
0927 02:36:08.969865 (+ 1262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:08.969868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:08.969925 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:08.969996 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:36:08.971212 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:08.971740 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:08.972838 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:08.973125 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:08.973214 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:08.975558 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:36:08.975562 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:36:08.975564 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:36:08.975588 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:36:08.976842 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:08.977306 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:08.977309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:08.977311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:08.977383 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:08.977688 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:08.977691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:08.977693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:08.977866 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:36:08.977955 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:08.978391 (+ 436us) server_negotiation.cc:299] Negotiation successful
0927 02:36:08.978511 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 02:36:08.979370 14335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:08.968070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.1:38441)
0927 02:36:08.968469 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:08.968507 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:08.969667 (+ 1160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:08.970153 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:36:08.970160 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:36:08.970550 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:36:08.971081 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:08.971094 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:08.971853 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:08.971855 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:08.972722 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:08.972732 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:08.973259 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:08.973263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:08.973414 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:08.973924 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:36:08.973939 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:36:08.975407 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:36:08.976966 (+ 1559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:08.976971 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:08.976985 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:08.977216 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:08.977462 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:08.977464 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:08.977466 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:08.977604 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:08.977950 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:36:08.977956 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:36:08.978182 (+ 226us) client_negotiation.cc:769] Sending connection context
0927 02:36:08.978336 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 02:36:08.978634 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":109,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 02:36:08.980567 14333 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36259
I20260927 02:36:08.980786 14333 heartbeater.cc:464] Registering TS with master...
I20260927 02:36:08.981307 14333 heartbeater.cc:511] Master 127.0.98.62:36259 requested a full tablet report, sending...
I20260927 02:36:08.982826 14142 ts_manager.cc:194] Registered new tserver with Master: d978229ba60143278b1226ab8132033d (127.0.98.1:45151)
I20260927 02:36:08.983913 14142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38441
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:36:08.993673 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36259
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:36:09.097864 14338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:36:09.098090 14338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:36:09.098147 14338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:36:09.102054 14338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:36:09.102104 14338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:36:09.102180 14338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:36:09.107200 14338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:36:09.108223 14338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:36:09.109215 14338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:36:09.119645 14343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:36:09.119711 14344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:36:09.119714 14346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:36:09.120051 14338 server_base.cc:1034] running on GCE node
I20260927 02:36:09.120384 14338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:36:09.120995 14338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:36:09.122156 14338 hybrid_clock.cc:648] HybridClock initialized: now 1790476569122131 us; error 39 us; skew 500 ppm
Sep 27 02:36:09 dist-test-slave-grlk krb5kdc[14097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:36:09.124673 14338 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:36:09.125561 14338 webserver.cc:460] Webserver started at http://127.0.98.2:40601/ using document root <none> and password file <none>
I20260927 02:36:09.126098 14338 fs_manager.cc:359] Metadata directory not provided
I20260927 02:36:09.126163 14338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:36:09.126343 14338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:36:09.128363 14338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-1/data/instance:
uuid: "64a55ced77b1438e87fbb5e5feb28738"
format_stamp: "Formatted at 2026-09-27 02:36:09 on dist-test-slave-grlk"
server_key: "7bb7680e522f3ba3c3b126c4092962a7"
server_key_iv: "c2d0b22f8f814c8396a18878bcbd04dd"
server_key_version: "encryptionkey@0"
I20260927 02:36:09.128885 14338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-1/wal/instance:
uuid: "64a55ced77b1438e87fbb5e5feb28738"
format_stamp: "Formatted at 2026-09-27 02:36:09 on dist-test-slave-grlk"
server_key: "7bb7680e522f3ba3c3b126c4092962a7"
server_key_iv: "c2d0b22f8f814c8396a18878bcbd04dd"
server_key_version: "encryptionkey@0"
I20260927 02:36:09.132386 14338 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 02:36:09.134884 14353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:09.136059 14338 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 02:36:09.136163 14338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-1/wal
uuid: "64a55ced77b1438e87fbb5e5feb28738"
format_stamp: "Formatted at 2026-09-27 02:36:09 on dist-test-slave-grlk"
server_key: "7bb7680e522f3ba3c3b126c4092962a7"
server_key_iv: "c2d0b22f8f814c8396a18878bcbd04dd"
server_key_version: "encryptionkey@0"
I20260927 02:36:09.136260 14338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:36:09.149832 14338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:36:09.152426 14338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:36:09.152587 14338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:36:09.153162 14338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:36:09.153915 14338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:36:09.153961 14338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:09.154013 14338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:36:09.154047 14338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:09.162719 14338 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34453
I20260927 02:36:09.162755 14466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34453 every 8 connection(s)
I20260927 02:36:09.163563 14338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:36:09.169065 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 14338
I20260927 02:36:09.169183 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:36:09.169467 392 external_mini_cluster.cc:1428] Setting key 519d422478051189e99b0cee2303488d
Sep 27 02:36:09 dist-test-slave-grlk krb5kdc[14097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:36:09.177081 14181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:09.165287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55551 (local address 127.0.98.62:36259)
0927 02:36:09.165486 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:09.165490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:09.166256 (+ 766us) server_negotiation.cc:407] Connection header received
0927 02:36:09.167410 (+ 1154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:09.167413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:09.167457 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:09.167527 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:36:09.168562 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:09.169112 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:09.170536 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:09.170886 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:09.170966 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:09.173746 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:36:09.173749 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:36:09.173751 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:36:09.173774 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:36:09.175098 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:09.175659 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:09.175662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:09.175663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:09.175729 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:09.176074 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:09.176076 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:09.176077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:09.176226 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:36:09.176292 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:09.176838 (+ 546us) server_negotiation.cc:299] Negotiation successful
0927 02:36:09.176925 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 02:36:09.178117 14469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:09.165691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.2:55551)
0927 02:36:09.166110 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:09.166150 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:09.167234 (+ 1084us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:09.167664 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:36:09.167670 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:36:09.167998 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:36:09.168451 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:09.168464 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:09.169244 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:09.169247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:09.170424 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:09.170434 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:09.171036 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:09.171053 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:09.171242 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:09.171788 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:36:09.171808 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:36:09.173601 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:36:09.175209 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:09.175216 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:09.175229 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:09.175569 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:09.175813 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:09.175816 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:09.175818 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:09.175991 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:09.176312 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:36:09.176318 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:36:09.176612 (+ 294us) client_negotiation.cc:769] Sending connection context
0927 02:36:09.176818 (+ 206us) client_negotiation.cc:240] Negotiation successful
0927 02:36:09.177156 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":115,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:36:09.179167 14467 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36259
I20260927 02:36:09.179374 14467 heartbeater.cc:464] Registering TS with master...
I20260927 02:36:09.179841 14467 heartbeater.cc:511] Master 127.0.98.62:36259 requested a full tablet report, sending...
I20260927 02:36:09.180781 14142 ts_manager.cc:194] Registered new tserver with Master: 64a55ced77b1438e87fbb5e5feb28738 (127.0.98.2:34453)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 02:36:09.181471 14142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55551
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:36:09.195339 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36259
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:36:09.300658 14472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:36:09.300948 14472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:36:09.301014 14472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:36:09.304956 14472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:36:09.305008 14472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:36:09.305088 14472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:36:09.310125 14472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:36:09.311133 14472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:36:09.312093 14472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:36:09.321214 14477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:36:09.321224 14478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:36:09.321257 14480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:36:09.321542 14472 server_base.cc:1034] running on GCE node
I20260927 02:36:09.321924 14472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:36:09.322531 14472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:36:09.323690 14472 hybrid_clock.cc:648] HybridClock initialized: now 1790476569323674 us; error 27 us; skew 500 ppm
Sep 27 02:36:09 dist-test-slave-grlk krb5kdc[14097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:36:09.326344 14472 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:36:09.327186 14472 webserver.cc:460] Webserver started at http://127.0.98.3:42463/ using document root <none> and password file <none>
I20260927 02:36:09.327704 14472 fs_manager.cc:359] Metadata directory not provided
I20260927 02:36:09.327771 14472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:36:09.327981 14472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:36:09.330034 14472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-2/data/instance:
uuid: "0b950ed19247482b9e0ff9b7a7d9eac1"
format_stamp: "Formatted at 2026-09-27 02:36:09 on dist-test-slave-grlk"
server_key: "96a4282c3ae44acfb5d64bd5a83d3b57"
server_key_iv: "bcddd3a4e526610621bd2bb38e779a47"
server_key_version: "encryptionkey@0"
I20260927 02:36:09.330600 14472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0b950ed19247482b9e0ff9b7a7d9eac1"
format_stamp: "Formatted at 2026-09-27 02:36:09 on dist-test-slave-grlk"
server_key: "96a4282c3ae44acfb5d64bd5a83d3b57"
server_key_iv: "bcddd3a4e526610621bd2bb38e779a47"
server_key_version: "encryptionkey@0"
I20260927 02:36:09.334314 14472 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:36:09.336565 14487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:09.337726 14472 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:36:09.337838 14472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-2/wal
uuid: "0b950ed19247482b9e0ff9b7a7d9eac1"
format_stamp: "Formatted at 2026-09-27 02:36:09 on dist-test-slave-grlk"
server_key: "96a4282c3ae44acfb5d64bd5a83d3b57"
server_key_iv: "bcddd3a4e526610621bd2bb38e779a47"
server_key_version: "encryptionkey@0"
I20260927 02:36:09.337941 14472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:36:09.352818 14472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:36:09.355357 14472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:36:09.355516 14472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:36:09.356038 14472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:36:09.356789 14472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:36:09.356835 14472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:09.356894 14472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:36:09.356920 14472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:09.365378 14472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39691
I20260927 02:36:09.365411 14600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39691 every 8 connection(s)
I20260927 02:36:09.366240 14472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:36:09.370802 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 14472
I20260927 02:36:09.370913 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:36:09.371160 392 external_mini_cluster.cc:1428] Setting key bc8e020610ce60e59ffc61ff8217117d
Sep 27 02:36:09 dist-test-slave-grlk krb5kdc[14097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:36:09.378253 14181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:09.368045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59495 (local address 127.0.98.62:36259)
0927 02:36:09.368182 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:09.368186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:09.368997 (+ 811us) server_negotiation.cc:407] Connection header received
0927 02:36:09.370225 (+ 1228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:09.370228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:09.370275 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:09.370345 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:36:09.371451 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:09.372140 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:09.373258 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:09.373474 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:09.373563 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:09.375607 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:36:09.375610 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:36:09.375612 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:36:09.375635 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:36:09.376705 (+ 1070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:09.377164 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:09.377166 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:09.377168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:09.377236 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:09.377466 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:09.377468 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:09.377469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:09.377613 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:36:09.377677 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:09.378053 (+ 376us) server_negotiation.cc:299] Negotiation successful
0927 02:36:09.378142 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 02:36:09.378938 14603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:09.368432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.3:59495)
0927 02:36:09.368846 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:09.368884 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:09.370049 (+ 1165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:09.370484 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:36:09.370492 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:36:09.370810 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:36:09.371317 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:09.371332 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:09.372239 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:09.372242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:09.373140 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:09.373149 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:09.373588 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:09.373591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:09.373718 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:09.374196 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:36:09.374210 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:36:09.375466 (+ 1256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:36:09.376795 (+ 1329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:09.376800 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:09.376809 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:09.377094 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:09.377300 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:09.377302 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:09.377303 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:09.377412 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:09.377674 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:36:09.377679 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:36:09.377869 (+ 190us) client_negotiation.cc:769] Sending connection context
0927 02:36:09.378010 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 02:36:09.378236 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":108,"threads_started":1}
I20260927 02:36:09.379869 14601 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36259
I20260927 02:36:09.380079 14601 heartbeater.cc:464] Registering TS with master...
I20260927 02:36:09.380494 14601 heartbeater.cc:511] Master 127.0.98.62:36259 requested a full tablet report, sending...
I20260927 02:36:09.381412 14142 ts_manager.cc:194] Registered new tserver with Master: 0b950ed19247482b9e0ff9b7a7d9eac1 (127.0.98.3:39691)
I20260927 02:36:09.381942 14142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59495
I20260927 02:36:09.384326 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:36:09.392424 14181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:09.386130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55140 (local address 127.0.98.62:36259)
0927 02:36:09.386271 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:09.386276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:09.386414 (+ 138us) server_negotiation.cc:407] Connection header received
0927 02:36:09.386495 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:09.386497 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:09.386533 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:09.386603 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:36:09.387190 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:09.387672 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:09.388403 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:09.388594 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:09.388665 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:09.389829 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:36:09.389832 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:36:09.389834 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:36:09.389867 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:36:09.391111 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:09.391502 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:09.391504 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:09.391506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:09.391581 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:09.391778 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:09.391781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:09.391782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:09.391964 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:36:09.392075 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:09.392202 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 02:36:09.392282 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 02:36:09.394711 14142 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55140:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:36:09.397492 14142 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:36:09.410823 14612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:09.404988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39691 (local address 127.0.0.1:43684)
0927 02:36:09.405586 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:09.405632 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:09.405752 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:09.406795 (+ 1043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:36:09.406800 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:36:09.406821 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:36:09.406976 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:09.406982 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:09.408017 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:09.408020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:09.409276 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:09.409293 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:09.410308 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:09.410311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:09.410411 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:09.410446 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 02:36:09.410548 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 02:36:09.410677 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"mutex_wait_us":69,"thread_start_us":79,"threads_started":1}
I20260927 02:36:09.410831 14616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:09.405660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43684 (local address 127.0.98.3:39691)
0927 02:36:09.406347 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:09.406359 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:09.406380 (+ 21us) server_negotiation.cc:407] Connection header received
0927 02:36:09.406449 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:09.406454 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:09.406643 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:09.406777 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:36:09.407095 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:09.407885 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:09.409406 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:09.410196 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:09.410288 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:09.410442 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:09.410575 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 02:36:09.410675 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":143,"threads_started":1}
I20260927 02:36:09.411293 14615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:09.405135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34453 (local address 127.0.0.1:46108)
0927 02:36:09.406082 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:09.406094 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:09.406188 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:09.406866 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:36:09.406869 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:36:09.406886 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:36:09.407001 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:09.407007 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:09.408459 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:09.408462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:09.409724 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:09.409732 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:09.410899 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:09.410903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:09.410998 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:09.411020 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 02:36:09.411092 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 02:36:09.411160 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":887,"mutex_wait_us":286,"thread_start_us":97,"threads_started":1}
I20260927 02:36:09.411366 14614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:09.405305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46108 (local address 127.0.98.2:34453)
0927 02:36:09.405847 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:09.405860 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:09.406277 (+ 417us) server_negotiation.cc:407] Connection header received
0927 02:36:09.406356 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:09.406360 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:09.406559 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:09.406708 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:36:09.407117 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:09.408314 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:09.409831 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:09.410596 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:09.410702 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:09.410838 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:09.411131 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 02:36:09.411229 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":145,"threads_started":1}
I20260927 02:36:09.411621 14617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:09.406057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50134 (local address 127.0.98.1:45151)
0927 02:36:09.406518 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:09.406529 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:09.406662 (+ 133us) server_negotiation.cc:407] Connection header received
0927 02:36:09.406770 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:09.406774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:09.406927 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:09.407084 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:36:09.407226 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:09.408314 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:09.410031 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:09.410803 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:09.410897 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:09.411022 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:09.411380 (+ 358us) server_negotiation.cc:299] Negotiation successful
0927 02:36:09.411493 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":120,"threads_started":1}
I20260927 02:36:09.411782 14613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:09.405689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45151 (local address 127.0.0.1:50134)
0927 02:36:09.406571 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:09.406583 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:09.406671 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:09.407063 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:36:09.407065 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:36:09.407083 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:36:09.407157 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:09.407162 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:09.408480 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:09.408483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:09.409907 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:09.409917 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:09.411077 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:09.411081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:09.411156 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:09.411174 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 02:36:09.411233 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 02:36:09.411285 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":826,"mutex_wait_us":50,"thread_start_us":79,"threads_started":1}
I20260927 02:36:09.414041 14535 tablet_service.cc:1467] Processing CreateTablet for tablet fa839b6e084842b78ee07bfd820649a3 (DEFAULT_TABLE table=test-table [id=3f42e309bd844ee8bdb0484d82b55f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:36:09.414352 14267 tablet_service.cc:1467] Processing CreateTablet for tablet fa839b6e084842b78ee07bfd820649a3 (DEFAULT_TABLE table=test-table [id=3f42e309bd844ee8bdb0484d82b55f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:36:09.414352 14401 tablet_service.cc:1467] Processing CreateTablet for tablet fa839b6e084842b78ee07bfd820649a3 (DEFAULT_TABLE table=test-table [id=3f42e309bd844ee8bdb0484d82b55f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:36:09.415074 14535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa839b6e084842b78ee07bfd820649a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:36:09.415329 14267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa839b6e084842b78ee07bfd820649a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:36:09.415329 14401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa839b6e084842b78ee07bfd820649a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:36:09.421005 14618 tablet_bootstrap.cc:492] T fa839b6e084842b78ee07bfd820649a3 P 0b950ed19247482b9e0ff9b7a7d9eac1: Bootstrap starting.
I20260927 02:36:09.422051 14619 tablet_bootstrap.cc:492] T fa839b6e084842b78ee07bfd820649a3 P d978229ba60143278b1226ab8132033d: Bootstrap starting.
I20260927 02:36:09.423017 14620 tablet_bootstrap.cc:492] T fa839b6e084842b78ee07bfd820649a3 P 64a55ced77b1438e87fbb5e5feb28738: Bootstrap starting.
I20260927 02:36:09.423188 14618 tablet_bootstrap.cc:654] T fa839b6e084842b78ee07bfd820649a3 P 0b950ed19247482b9e0ff9b7a7d9eac1: Neither blocks nor log segments found. Creating new log.
I20260927 02:36:09.424060 14619 tablet_bootstrap.cc:654] T fa839b6e084842b78ee07bfd820649a3 P d978229ba60143278b1226ab8132033d: Neither blocks nor log segments found. Creating new log.
I20260927 02:36:09.424386 14618 log.cc:824] T fa839b6e084842b78ee07bfd820649a3 P 0b950ed19247482b9e0ff9b7a7d9eac1: Log is configured to *not* fsync() on all Append() calls
I20260927 02:36:09.424948 14620 tablet_bootstrap.cc:654] T fa839b6e084842b78ee07bfd820649a3 P 64a55ced77b1438e87fbb5e5feb28738: Neither blocks nor log segments found. Creating new log.
I20260927 02:36:09.425182 14619 log.cc:824] T fa839b6e084842b78ee07bfd820649a3 P d978229ba60143278b1226ab8132033d: Log is configured to *not* fsync() on all Append() calls
I20260927 02:36:09.425746 14620 log.cc:824] T fa839b6e084842b78ee07bfd820649a3 P 64a55ced77b1438e87fbb5e5feb28738: Log is configured to *not* fsync() on all Append() calls
I20260927 02:36:09.426589 14618 tablet_bootstrap.cc:492] T fa839b6e084842b78ee07bfd820649a3 P 0b950ed19247482b9e0ff9b7a7d9eac1: No bootstrap required, opened a new log
I20260927 02:36:09.426776 14618 ts_tablet_manager.cc:1397] T fa839b6e084842b78ee07bfd820649a3 P 0b950ed19247482b9e0ff9b7a7d9eac1: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 02:36:09.426980 14619 tablet_bootstrap.cc:492] T fa839b6e084842b78ee07bfd820649a3 P d978229ba60143278b1226ab8132033d: No bootstrap required, opened a new log
I20260927 02:36:09.427150 14619 ts_tablet_manager.cc:1397] T fa839b6e084842b78ee07bfd820649a3 P d978229ba60143278b1226ab8132033d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 02:36:09.427373 14620 tablet_bootstrap.cc:492] T fa839b6e084842b78ee07bfd820649a3 P 64a55ced77b1438e87fbb5e5feb28738: No bootstrap required, opened a new log
I20260927 02:36:09.427592 14620 ts_tablet_manager.cc:1397] T fa839b6e084842b78ee07bfd820649a3 P 64a55ced77b1438e87fbb5e5feb28738: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 02:36:09.430330 14620 raft_consensus.cc:348] T fa839b6e084842b78ee07bfd820649a3 P 64a55ced77b1438e87fbb5e5feb28738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b950ed19247482b9e0ff9b7a7d9eac1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39691 } } peers { permanent_uuid: "64a55ced77b1438e87fbb5e5feb28738" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34453 } } peers { permanent_uuid: "d978229ba60143278b1226ab8132033d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45151 } }
I20260927 02:36:09.430554 14620 raft_consensus.cc:374] T fa839b6e084842b78ee07bfd820649a3 P 64a55ced77b1438e87fbb5e5feb28738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:36:09.430624 14620 raft_consensus.cc:729] T fa839b6e084842b78ee07bfd820649a3 P 64a55ced77b1438e87fbb5e5feb28738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64a55ced77b1438e87fbb5e5feb28738, State: Initialized, Role: FOLLOWER
I20260927 02:36:09.430471 14618 raft_consensus.cc:348] T fa839b6e084842b78ee07bfd820649a3 P 0b950ed19247482b9e0ff9b7a7d9eac1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b950ed19247482b9e0ff9b7a7d9eac1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39691 } } peers { permanent_uuid: "64a55ced77b1438e87fbb5e5feb28738" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34453 } } peers { permanent_uuid: "d978229ba60143278b1226ab8132033d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45151 } }
I20260927 02:36:09.430763 14618 raft_consensus.cc:374] T fa839b6e084842b78ee07bfd820649a3 P 0b950ed19247482b9e0ff9b7a7d9eac1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:36:09.430634 14619 raft_consensus.cc:348] T fa839b6e084842b78ee07bfd820649a3 P d978229ba60143278b1226ab8132033d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b950ed19247482b9e0ff9b7a7d9eac1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39691 } } peers { permanent_uuid: "64a55ced77b1438e87fbb5e5feb28738" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34453 } } peers { permanent_uuid: "d978229ba60143278b1226ab8132033d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45151 } }
I20260927 02:36:09.430819 14618 raft_consensus.cc:729] T fa839b6e084842b78ee07bfd820649a3 P 0b950ed19247482b9e0ff9b7a7d9eac1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b950ed19247482b9e0ff9b7a7d9eac1, State: Initialized, Role: FOLLOWER
I20260927 02:36:09.430826 14619 raft_consensus.cc:374] T fa839b6e084842b78ee07bfd820649a3 P d978229ba60143278b1226ab8132033d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:36:09.430876 14619 raft_consensus.cc:729] T fa839b6e084842b78ee07bfd820649a3 P d978229ba60143278b1226ab8132033d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d978229ba60143278b1226ab8132033d, State: Initialized, Role: FOLLOWER
I20260927 02:36:09.431140 14620 consensus_queue.cc:260] T fa839b6e084842b78ee07bfd820649a3 P 64a55ced77b1438e87fbb5e5feb28738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b950ed19247482b9e0ff9b7a7d9eac1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39691 } } peers { permanent_uuid: "64a55ced77b1438e87fbb5e5feb28738" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34453 } } peers { permanent_uuid: "d978229ba60143278b1226ab8132033d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45151 } }
I20260927 02:36:09.431263 14619 consensus_queue.cc:260] T fa839b6e084842b78ee07bfd820649a3 P d978229ba60143278b1226ab8132033d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b950ed19247482b9e0ff9b7a7d9eac1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39691 } } peers { permanent_uuid: "64a55ced77b1438e87fbb5e5feb28738" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34453 } } peers { permanent_uuid: "d978229ba60143278b1226ab8132033d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45151 } }
I20260927 02:36:09.431326 14618 consensus_queue.cc:260] T fa839b6e084842b78ee07bfd820649a3 P 0b950ed19247482b9e0ff9b7a7d9eac1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b950ed19247482b9e0ff9b7a7d9eac1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39691 } } peers { permanent_uuid: "64a55ced77b1438e87fbb5e5feb28738" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34453 } } peers { permanent_uuid: "d978229ba60143278b1226ab8132033d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45151 } }
I20260927 02:36:09.432006 14333 heartbeater.cc:503] Master 127.0.98.62:36259 was elected leader, sending a full tablet report...
I20260927 02:36:09.432166 14620 ts_tablet_manager.cc:1428] T fa839b6e084842b78ee07bfd820649a3 P 64a55ced77b1438e87fbb5e5feb28738: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 02:36:09.432264 14619 ts_tablet_manager.cc:1428] T fa839b6e084842b78ee07bfd820649a3 P d978229ba60143278b1226ab8132033d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 02:36:09.432482 14467 heartbeater.cc:503] Master 127.0.98.62:36259 was elected leader, sending a full tablet report...
I20260927 02:36:09.433171 14618 ts_tablet_manager.cc:1428] T fa839b6e084842b78ee07bfd820649a3 P 0b950ed19247482b9e0ff9b7a7d9eac1: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 02:36:09.433465 14601 heartbeater.cc:503] Master 127.0.98.62:36259 was elected leader, sending a full tablet report...
W20260927 02:36:09.467173 14334 tablet.cc:2369] T fa839b6e084842b78ee07bfd820649a3 P d978229ba60143278b1226ab8132033d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:36:09.617143 14602 tablet.cc:2369] T fa839b6e084842b78ee07bfd820649a3 P 0b950ed19247482b9e0ff9b7a7d9eac1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:36:09.645898 14624 raft_consensus.cc:482] T fa839b6e084842b78ee07bfd820649a3 P 64a55ced77b1438e87fbb5e5feb28738 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:36:09.646071 14624 raft_consensus.cc:504] T fa839b6e084842b78ee07bfd820649a3 P 64a55ced77b1438e87fbb5e5feb28738 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b950ed19247482b9e0ff9b7a7d9eac1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39691 } } peers { permanent_uuid: "64a55ced77b1438e87fbb5e5feb28738" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34453 } } peers { permanent_uuid: "d978229ba60143278b1226ab8132033d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45151 } }
I20260927 02:36:09.647073 14624 leader_election.cc:288] T fa839b6e084842b78ee07bfd820649a3 P 64a55ced77b1438e87fbb5e5feb28738 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b950ed19247482b9e0ff9b7a7d9eac1 (127.0.98.3:39691), d978229ba60143278b1226ab8132033d (127.0.98.1:45151)
I20260927 02:36:09.650636 14617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:09.647386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40535 (local address 127.0.98.1:45151)
0927 02:36:09.647524 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:09.647527 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:09.647671 (+ 144us) server_negotiation.cc:407] Connection header received
0927 02:36:09.647816 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:09.647819 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:09.647860 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:09.647921 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:36:09.648158 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:09.648649 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:09.649711 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:09.650122 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:09.650221 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:09.650281 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:09.650443 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 02:36:09.650518 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 02:36:09.650656 14469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:09.647297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45151 (local address 127.0.98.2:40535)
0927 02:36:09.647605 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:09.647616 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:09.647723 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:09.647937 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:36:09.647940 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:36:09.647953 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:36:09.648069 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:09.648074 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:09.648744 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:09.648747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:09.649598 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:09.649607 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:09.650232 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:09.650236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:09.650338 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:09.650358 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 02:36:09.650422 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 02:36:09.650490 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"spinlock_wait_cycles":24320,"thread_start_us":66,"threads_started":1}
I20260927 02:36:09.650679 14616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:09.647360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52675 (local address 127.0.98.3:39691)
0927 02:36:09.647482 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:09.647487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:09.647672 (+ 185us) server_negotiation.cc:407] Connection header received
0927 02:36:09.647834 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:09.647837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:09.647882 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:09.647940 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:36:09.648194 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:09.648714 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:09.649772 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:09.650169 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:09.650261 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:09.650318 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:09.650488 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 02:36:09.650562 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 02:36:09.650698 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:09.647268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39691 (local address 127.0.98.2:52675)
0927 02:36:09.647588 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:09.647599 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:09.647743 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:09.647963 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:36:09.647965 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:36:09.647980 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:36:09.648102 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:09.648107 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:09.648810 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:09.648814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:09.649653 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:09.649662 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:09.650277 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:09.650280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:09.650380 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:09.650401 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 02:36:09.650461 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 02:36:09.650527 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":1,"spinlock_wait_cycles":11776}
I20260927 02:36:09.651345 14287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa839b6e084842b78ee07bfd820649a3" candidate_uuid: "64a55ced77b1438e87fbb5e5feb28738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d978229ba60143278b1226ab8132033d" is_pre_election: true
I20260927 02:36:09.651346 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa839b6e084842b78ee07bfd820649a3" candidate_uuid: "64a55ced77b1438e87fbb5e5feb28738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b950ed19247482b9e0ff9b7a7d9eac1" is_pre_election: true
I20260927 02:36:09.651625 14287 raft_consensus.cc:2454] T fa839b6e084842b78ee07bfd820649a3 P d978229ba60143278b1226ab8132033d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64a55ced77b1438e87fbb5e5feb28738 in term 0.
I20260927 02:36:09.651625 14555 raft_consensus.cc:2454] T fa839b6e084842b78ee07bfd820649a3 P 0b950ed19247482b9e0ff9b7a7d9eac1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64a55ced77b1438e87fbb5e5feb28738 in term 0.
I20260927 02:36:09.652048 14354 leader_election.cc:302] T fa839b6e084842b78ee07bfd820649a3 P 64a55ced77b1438e87fbb5e5feb28738 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64a55ced77b1438e87fbb5e5feb28738, d978229ba60143278b1226ab8132033d; no voters:
I20260927 02:36:09.652285 14624 raft_consensus.cc:2781] T fa839b6e084842b78ee07bfd820649a3 P 64a55ced77b1438e87fbb5e5feb28738 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:36:09.652365 14624 raft_consensus.cc:482] T fa839b6e084842b78ee07bfd820649a3 P 64a55ced77b1438e87fbb5e5feb28738 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:36:09.652410 14624 raft_consensus.cc:3037] T fa839b6e084842b78ee07bfd820649a3 P 64a55ced77b1438e87fbb5e5feb28738 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:36:09.653370 14624 raft_consensus.cc:504] T fa839b6e084842b78ee07bfd820649a3 P 64a55ced77b1438e87fbb5e5feb28738 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b950ed19247482b9e0ff9b7a7d9eac1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39691 } } peers { permanent_uuid: "64a55ced77b1438e87fbb5e5feb28738" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34453 } } peers { permanent_uuid: "d978229ba60143278b1226ab8132033d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45151 } }
I20260927 02:36:09.653648 14624 leader_election.cc:288] T fa839b6e084842b78ee07bfd820649a3 P 64a55ced77b1438e87fbb5e5feb28738 [CANDIDATE]: Term 1 election: Requested vote from peers 0b950ed19247482b9e0ff9b7a7d9eac1 (127.0.98.3:39691), d978229ba60143278b1226ab8132033d (127.0.98.1:45151)
I20260927 02:36:09.653918 14555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa839b6e084842b78ee07bfd820649a3" candidate_uuid: "64a55ced77b1438e87fbb5e5feb28738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b950ed19247482b9e0ff9b7a7d9eac1"
I20260927 02:36:09.653992 14287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa839b6e084842b78ee07bfd820649a3" candidate_uuid: "64a55ced77b1438e87fbb5e5feb28738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d978229ba60143278b1226ab8132033d"
I20260927 02:36:09.654023 14555 raft_consensus.cc:3037] T fa839b6e084842b78ee07bfd820649a3 P 0b950ed19247482b9e0ff9b7a7d9eac1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:36:09.654119 14287 raft_consensus.cc:3037] T fa839b6e084842b78ee07bfd820649a3 P d978229ba60143278b1226ab8132033d [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:36:09.654970 14555 raft_consensus.cc:2454] T fa839b6e084842b78ee07bfd820649a3 P 0b950ed19247482b9e0ff9b7a7d9eac1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64a55ced77b1438e87fbb5e5feb28738 in term 1.
I20260927 02:36:09.655031 14287 raft_consensus.cc:2454] T fa839b6e084842b78ee07bfd820649a3 P d978229ba60143278b1226ab8132033d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64a55ced77b1438e87fbb5e5feb28738 in term 1.
I20260927 02:36:09.655241 14355 leader_election.cc:302] T fa839b6e084842b78ee07bfd820649a3 P 64a55ced77b1438e87fbb5e5feb28738 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b950ed19247482b9e0ff9b7a7d9eac1, 64a55ced77b1438e87fbb5e5feb28738; no voters:
I20260927 02:36:09.655387 14624 raft_consensus.cc:2781] T fa839b6e084842b78ee07bfd820649a3 P 64a55ced77b1438e87fbb5e5feb28738 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:36:09.655582 14624 raft_consensus.cc:686] T fa839b6e084842b78ee07bfd820649a3 P 64a55ced77b1438e87fbb5e5feb28738 [term 1 LEADER]: Becoming Leader. State: Replica: 64a55ced77b1438e87fbb5e5feb28738, State: Running, Role: LEADER
I20260927 02:36:09.655865 14624 consensus_queue.cc:237] T fa839b6e084842b78ee07bfd820649a3 P 64a55ced77b1438e87fbb5e5feb28738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b950ed19247482b9e0ff9b7a7d9eac1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39691 } } peers { permanent_uuid: "64a55ced77b1438e87fbb5e5feb28738" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34453 } } peers { permanent_uuid: "d978229ba60143278b1226ab8132033d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45151 } }
I20260927 02:36:09.658701 14141 catalog_manager.cc:5495] T fa839b6e084842b78ee07bfd820649a3 P 64a55ced77b1438e87fbb5e5feb28738 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64a55ced77b1438e87fbb5e5feb28738 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "64a55ced77b1438e87fbb5e5feb28738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b950ed19247482b9e0ff9b7a7d9eac1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64a55ced77b1438e87fbb5e5feb28738" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d978229ba60143278b1226ab8132033d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45151 } health_report { overall_health: UNKNOWN } } }
W20260927 02:36:09.667457 14468 tablet.cc:2369] T fa839b6e084842b78ee07bfd820649a3 P 64a55ced77b1438e87fbb5e5feb28738: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:36:09.704351 14614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:09.701399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46120 (local address 127.0.98.2:34453)
0927 02:36:09.701523 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:09.701527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:09.701540 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:36:09.701659 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:09.701661 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:09.701716 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:09.701798 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:36:09.701960 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:09.702436 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:09.703169 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:09.703366 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:09.703430 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:09.703650 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:36:09.704047 (+ 397us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:36:09.704129 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:09.704171 (+ 42us) server_negotiation.cc:299] Negotiation successful
0927 02:36:09.704218 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 02:36:09.709830 14287 raft_consensus.cc:1261] T fa839b6e084842b78ee07bfd820649a3 P d978229ba60143278b1226ab8132033d [term 1 FOLLOWER]: Refusing update from remote peer 64a55ced77b1438e87fbb5e5feb28738: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:36:09.709830 14555 raft_consensus.cc:1261] T fa839b6e084842b78ee07bfd820649a3 P 0b950ed19247482b9e0ff9b7a7d9eac1 [term 1 FOLLOWER]: Refusing update from remote peer 64a55ced77b1438e87fbb5e5feb28738: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:36:09.710587 14624 consensus_queue.cc:1035] T fa839b6e084842b78ee07bfd820649a3 P 64a55ced77b1438e87fbb5e5feb28738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d978229ba60143278b1226ab8132033d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:36:09.710788 14628 consensus_queue.cc:1035] T fa839b6e084842b78ee07bfd820649a3 P 64a55ced77b1438e87fbb5e5feb28738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b950ed19247482b9e0ff9b7a7d9eac1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:36:09.726913 14140 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:36:09.727072 14140 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:36:09.729447 14140 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e: Sending DeleteTablet for 3 replicas of tablet fa839b6e084842b78ee07bfd820649a3
I20260927 02:36:09.730294 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 14204
I20260927 02:36:09.730438 14535 tablet_service.cc:1514] Processing DeleteTablet for tablet fa839b6e084842b78ee07bfd820649a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:36:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43684
I20260927 02:36:09.730924 14401 tablet_service.cc:1514] Processing DeleteTablet for tablet fa839b6e084842b78ee07bfd820649a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:36:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46108
I20260927 02:36:09.731349 14642 tablet_replica.cc:330] T fa839b6e084842b78ee07bfd820649a3 P 64a55ced77b1438e87fbb5e5feb28738: stopping tablet replica
I20260927 02:36:09.731678 14642 raft_consensus.cc:2229] T fa839b6e084842b78ee07bfd820649a3 P 64a55ced77b1438e87fbb5e5feb28738 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:36:09.732053 14642 raft_consensus.cc:2258] T fa839b6e084842b78ee07bfd820649a3 P 64a55ced77b1438e87fbb5e5feb28738 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:36:09.733487 14642 ts_tablet_manager.cc:1905] T fa839b6e084842b78ee07bfd820649a3 P 64a55ced77b1438e87fbb5e5feb28738: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:36:09.736209 14642 ts_tablet_manager.cc:1918] T fa839b6e084842b78ee07bfd820649a3 P 64a55ced77b1438e87fbb5e5feb28738: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:36:09.736238 14641 tablet_replica.cc:330] T fa839b6e084842b78ee07bfd820649a3 P 0b950ed19247482b9e0ff9b7a7d9eac1: stopping tablet replica
I20260927 02:36:09.736346 14642 log.cc:1196] T fa839b6e084842b78ee07bfd820649a3 P 64a55ced77b1438e87fbb5e5feb28738: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-1/wal/wals/fa839b6e084842b78ee07bfd820649a3
W20260927 02:36:09.736449 14126 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45151 (error 108)
I20260927 02:36:09.736577 14641 raft_consensus.cc:2229] T fa839b6e084842b78ee07bfd820649a3 P 0b950ed19247482b9e0ff9b7a7d9eac1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:36:09.736663 14642 ts_tablet_manager.cc:1939] T fa839b6e084842b78ee07bfd820649a3 P 64a55ced77b1438e87fbb5e5feb28738: Deleting consensus metadata
I20260927 02:36:09.736899 14641 raft_consensus.cc:2258] T fa839b6e084842b78ee07bfd820649a3 P 0b950ed19247482b9e0ff9b7a7d9eac1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:36:09.737385 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 14338
I20260927 02:36:09.738555 14641 ts_tablet_manager.cc:1905] T fa839b6e084842b78ee07bfd820649a3 P 0b950ed19247482b9e0ff9b7a7d9eac1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:36:09.739275 14612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:09.738969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45151 (local address 127.0.0.1:50136)
0927 02:36:09.739130 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:09.739193 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260927 02:36:09.739452 14126 catalog_manager.cc:4584] TS d978229ba60143278b1226ab8132033d (127.0.98.1:45151): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fa839b6e084842b78ee07bfd820649a3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45151: connect: Connection refused (error 111)
I20260927 02:36:09.740863 14641 ts_tablet_manager.cc:1918] T fa839b6e084842b78ee07bfd820649a3 P 0b950ed19247482b9e0ff9b7a7d9eac1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:36:09.740978 14641 log.cc:1196] T fa839b6e084842b78ee07bfd820649a3 P 0b950ed19247482b9e0ff9b7a7d9eac1: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-2/wal/wals/fa839b6e084842b78ee07bfd820649a3
I20260927 02:36:09.741259 14641 ts_tablet_manager.cc:1939] T fa839b6e084842b78ee07bfd820649a3 P 0b950ed19247482b9e0ff9b7a7d9eac1: Deleting consensus metadata
I20260927 02:36:09.742352 14127 catalog_manager.cc:4841] TS 0b950ed19247482b9e0ff9b7a7d9eac1 (127.0.98.3:39691): tablet fa839b6e084842b78ee07bfd820649a3 (table test-table [id=3f42e309bd844ee8bdb0484d82b55f9f]) successfully deleted
I20260927 02:36:09.743103 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 14472
I20260927 02:36:09.744077 14612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:09.743809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34453 (local address 127.0.0.1:46128)
0927 02:36:09.743959 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:09.744003 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:34453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 02:36:09.747592 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 14110
I20260927 02:36:09.752736 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36259
--webserver_interface=127.0.98.62
--webserver_port=42597
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:36:09.858544 14644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:36:09.858800 14644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:36:09.858851 14644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:36:09.863092 14644 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:36:09.863195 14644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:36:09.863235 14644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:36:09.863260 14644 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:36:09.863281 14644 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:36:09.868732 14644 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:36:09.869788 14644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:36:09.870748 14644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:36:09.882501 14653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:36:09.882570 14644 server_base.cc:1034] running on GCE node
W20260927 02:36:09.882488 14650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:36:09.882501 14651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:36:09.883312 14644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:36:09.884323 14644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:36:09.885464 14644 hybrid_clock.cc:648] HybridClock initialized: now 1790476569885441 us; error 35 us; skew 500 ppm
Sep 27 02:36:09 dist-test-slave-grlk krb5kdc[14097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:36:09.887923 14644 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:36:09.888737 14644 webserver.cc:460] Webserver started at http://127.0.98.62:42597/ using document root <none> and password file <none>
I20260927 02:36:09.889281 14644 fs_manager.cc:359] Metadata directory not provided
I20260927 02:36:09.889356 14644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:36:09.892906 14644 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:36:09.894884 14660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:09.895934 14644 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:36:09.896050 14644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "b53ecc8c654d41c7a4e117f1ef1d345e"
format_stamp: "Formatted at 2026-09-27 02:36:08 on dist-test-slave-grlk"
server_key: "c22e67050e2607299641684c3b867e42"
server_key_iv: "eefafd85429342ad55b0acd8de8f4e9a"
server_key_version: "encryptionkey@0"
I20260927 02:36:09.896581 14644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:36:09.918092 14644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:36:09.920791 14644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:36:09.920993 14644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:36:09.927560 14644 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36259
I20260927 02:36:09.927827 14712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36259 every 8 connection(s)
I20260927 02:36:09.928588 14644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:36:09.932610 14713 sys_catalog.cc:263] Verifying existing consensus state
I20260927 02:36:09.934231 14713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e: Bootstrap starting.
I20260927 02:36:09.938314 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 14644
I20260927 02:36:09.938694 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45151
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36259
--webserver_port=43069
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:36:09.950284 14713 log.cc:824] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e: Log is configured to *not* fsync() on all Append() calls
I20260927 02:36:09.958573 14713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:36:09.958899 14713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e: Bootstrap complete.
I20260927 02:36:09.962718 14713 raft_consensus.cc:348] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53ecc8c654d41c7a4e117f1ef1d345e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36259 } }
I20260927 02:36:09.963099 14713 raft_consensus.cc:729] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b53ecc8c654d41c7a4e117f1ef1d345e, State: Initialized, Role: FOLLOWER
I20260927 02:36:09.963560 14713 consensus_queue.cc:260] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53ecc8c654d41c7a4e117f1ef1d345e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36259 } }
I20260927 02:36:09.963685 14713 raft_consensus.cc:388] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:36:09.963752 14713 raft_consensus.cc:482] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:36:09.963845 14713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [term 1 FOLLOWER]: Advancing to term 2
I20260927 02:36:09.965660 14713 raft_consensus.cc:504] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53ecc8c654d41c7a4e117f1ef1d345e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36259 } }
I20260927 02:36:09.965957 14713 leader_election.cc:302] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b53ecc8c654d41c7a4e117f1ef1d345e; no voters:
I20260927 02:36:09.966224 14713 leader_election.cc:288] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 02:36:09.966365 14717 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [term 2 FOLLOWER]: Leader election won for term 2
I20260927 02:36:09.966588 14717 raft_consensus.cc:686] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [term 2 LEADER]: Becoming Leader. State: Replica: b53ecc8c654d41c7a4e117f1ef1d345e, State: Running, Role: LEADER
I20260927 02:36:09.966876 14717 consensus_queue.cc:237] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53ecc8c654d41c7a4e117f1ef1d345e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36259 } }
I20260927 02:36:09.967303 14713 sys_catalog.cc:564] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:36:09.968257 14719 sys_catalog.cc:455] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b53ecc8c654d41c7a4e117f1ef1d345e. Latest consensus state: current_term: 2 leader_uuid: "b53ecc8c654d41c7a4e117f1ef1d345e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53ecc8c654d41c7a4e117f1ef1d345e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36259 } } }
I20260927 02:36:09.968286 14718 sys_catalog.cc:455] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b53ecc8c654d41c7a4e117f1ef1d345e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53ecc8c654d41c7a4e117f1ef1d345e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36259 } } }
I20260927 02:36:09.968369 14719 sys_catalog.cc:458] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [sys.catalog]: This master's current role is: LEADER
I20260927 02:36:09.968381 14718 sys_catalog.cc:458] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [sys.catalog]: This master's current role is: LEADER
I20260927 02:36:09.968684 14726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:36:09.973168 14726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:36:09.973745 14726 catalog_manager.cc:1259] Loaded cluster ID: f86887c5d3c2418f83ba2792ec8f45bf
I20260927 02:36:09.973799 14726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:36:09.975096 14726 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 02:36:09.979490 14726 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 02:36:09.984328 14726 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:36:09.985075 14726 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e: Loaded TSK: 0
I20260927 02:36:09.986224 14726 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 02:36:10.050074 14715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:36:10.050318 14715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:36:10.050360 14715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:36:10.054323 14715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:36:10.054381 14715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:36:10.054441 14715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:36:10.059595 14715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45151
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43069
--tserver_master_addrs=127.0.98.62:36259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:36:10.060554 14715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:36:10.061535 14715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:36:10.070452 14741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:36:10.070526 14740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:36:10.070453 14743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:36:10.070670 14715 server_base.cc:1034] running on GCE node
I20260927 02:36:10.071085 14715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:36:10.071691 14715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:36:10.072849 14715 hybrid_clock.cc:648] HybridClock initialized: now 1790476570072821 us; error 44 us; skew 500 ppm
Sep 27 02:36:10 dist-test-slave-grlk krb5kdc[14097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:36:10.075500 14715 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:36:10.076524 14715 webserver.cc:460] Webserver started at http://127.0.98.1:43069/ using document root <none> and password file <none>
I20260927 02:36:10.077224 14715 fs_manager.cc:359] Metadata directory not provided
I20260927 02:36:10.077333 14715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:36:10.082436 14715 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:36:10.085035 14750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:10.086496 14715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:36:10.086612 14715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-0/wal
uuid: "d978229ba60143278b1226ab8132033d"
format_stamp: "Formatted at 2026-09-27 02:36:08 on dist-test-slave-grlk"
server_key: "667717698431fe3ce035658e6d4e7f21"
server_key_iv: "96c13820116b71566586b6eeec4bc1f8"
server_key_version: "encryptionkey@0"
I20260927 02:36:10.087332 14715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:36:10.110636 14715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:36:10.114450 14715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:36:10.114640 14715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:36:10.115239 14715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:36:10.116504 14757 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 02:36:10.121647 14715 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 02:36:10.121729 14715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260927 02:36:10.121804 14715 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 02:36:10.123454 14715 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 02:36:10.123507 14715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 02:36:10.123869 14757 tablet_bootstrap.cc:492] T fa839b6e084842b78ee07bfd820649a3 P d978229ba60143278b1226ab8132033d: Bootstrap starting.
I20260927 02:36:10.132830 14715 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45151
I20260927 02:36:10.132882 14864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45151 every 8 connection(s)
I20260927 02:36:10.133906 14715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:36:10.134579 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 14715
I20260927 02:36:10.135071 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34453
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36259
--webserver_port=40601
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 02:36:10 dist-test-slave-grlk krb5kdc[14097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:36:10.147872 14757 log.cc:824] T fa839b6e084842b78ee07bfd820649a3 P d978229ba60143278b1226ab8132033d: Log is configured to *not* fsync() on all Append() calls
I20260927 02:36:10.154300 14868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:10.136021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.1:40351)
0927 02:36:10.136543 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:10.136597 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:10.137571 (+ 974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:10.138709 (+ 1138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:36:10.138717 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:36:10.139192 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:36:10.140016 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:10.140070 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:10.141841 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:10.141846 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:10.142717 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:10.142728 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:10.143447 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:10.143451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:10.143634 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:10.144233 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:36:10.144248 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:36:10.147589 (+ 3341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:36:10.150873 (+ 3284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:10.150879 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:10.150895 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:10.151234 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:10.151635 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:10.151638 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:10.151641 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:10.151803 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:10.152495 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:36:10.152502 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:36:10.152781 (+ 279us) client_negotiation.cc:769] Sending connection context
0927 02:36:10.152983 (+ 202us) client_negotiation.cc:240] Negotiation successful
0927 02:36:10.153307 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":136,"threads_started":1}
I20260927 02:36:10.156018 14867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:10.136014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40351 (local address 127.0.98.62:36259)
0927 02:36:10.137175 (+ 1161us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:10.137193 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:10.137239 (+ 46us) server_negotiation.cc:407] Connection header received
0927 02:36:10.137982 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:10.137993 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:10.138330 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:10.138639 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:36:10.140458 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:10.141646 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:10.142848 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:10.143305 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:10.143451 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:10.147817 (+ 4366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:36:10.147824 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:36:10.147840 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:36:10.147876 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:36:10.150724 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:10.151358 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:10.151364 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:10.151369 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:10.151540 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:10.151914 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:10.151918 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:10.151920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:10.152386 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:36:10.152513 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:10.154878 (+ 2365us) server_negotiation.cc:299] Negotiation successful
0927 02:36:10.155195 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1021,"thread_start_us":117,"threads_started":1}
I20260927 02:36:10.156271 14757 tablet_bootstrap.cc:492] T fa839b6e084842b78ee07bfd820649a3 P d978229ba60143278b1226ab8132033d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:36:10.156661 14757 tablet_bootstrap.cc:492] T fa839b6e084842b78ee07bfd820649a3 P d978229ba60143278b1226ab8132033d: Bootstrap complete.
I20260927 02:36:10.157681 14757 ts_tablet_manager.cc:1397] T fa839b6e084842b78ee07bfd820649a3 P d978229ba60143278b1226ab8132033d: Time spent bootstrapping tablet: real 0.034s user 0.025s sys 0.004s
I20260927 02:36:10.158306 14865 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36259
I20260927 02:36:10.158530 14865 heartbeater.cc:464] Registering TS with master...
I20260927 02:36:10.159045 14865 heartbeater.cc:511] Master 127.0.98.62:36259 requested a full tablet report, sending...
I20260927 02:36:10.160404 14757 raft_consensus.cc:348] T fa839b6e084842b78ee07bfd820649a3 P d978229ba60143278b1226ab8132033d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b950ed19247482b9e0ff9b7a7d9eac1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39691 } } peers { permanent_uuid: "64a55ced77b1438e87fbb5e5feb28738" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34453 } } peers { permanent_uuid: "d978229ba60143278b1226ab8132033d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45151 } }
I20260927 02:36:10.160843 14757 raft_consensus.cc:729] T fa839b6e084842b78ee07bfd820649a3 P d978229ba60143278b1226ab8132033d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d978229ba60143278b1226ab8132033d, State: Initialized, Role: FOLLOWER
I20260927 02:36:10.161281 14757 consensus_queue.cc:260] T fa839b6e084842b78ee07bfd820649a3 P d978229ba60143278b1226ab8132033d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b950ed19247482b9e0ff9b7a7d9eac1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39691 } } peers { permanent_uuid: "64a55ced77b1438e87fbb5e5feb28738" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34453 } } peers { permanent_uuid: "d978229ba60143278b1226ab8132033d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45151 } }
I20260927 02:36:10.161684 14677 ts_manager.cc:194] Registered new tserver with Master: d978229ba60143278b1226ab8132033d (127.0.98.1:45151)
I20260927 02:36:10.162132 14757 ts_tablet_manager.cc:1428] T fa839b6e084842b78ee07bfd820649a3 P d978229ba60143278b1226ab8132033d: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260927 02:36:10.165484 14677 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40351
I20260927 02:36:10.168237 14865 heartbeater.cc:503] Master 127.0.98.62:36259 was elected leader, sending a full tablet report...
Sep 27 02:36:10 dist-test-slave-grlk krb5kdc[14097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:36:10.174773 14872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:10.164788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45151 (local address 127.0.0.1:50138)
0927 02:36:10.165492 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:10.165524 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:10.165659 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:10.166076 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:36:10.166082 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:36:10.166491 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:36:10.166594 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:10.166600 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:10.167555 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:10.167558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:10.168273 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:10.168283 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:10.169011 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:10.169027 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:10.169138 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:10.169730 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:36:10.169744 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:36:10.171236 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:36:10.173097 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:10.173105 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:10.173110 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:10.173373 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:10.173621 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:10.173624 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:10.173626 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:10.173714 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:10.174132 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:36:10.174138 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:36:10.174254 (+ 116us) client_negotiation.cc:769] Sending connection context
0927 02:36:10.174416 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 02:36:10.174597 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":101,"threads_started":1}
I20260927 02:36:10.174795 14873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:10.164891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50138 (local address 127.0.98.1:45151)
0927 02:36:10.165281 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:10.165291 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:10.165663 (+ 372us) server_negotiation.cc:407] Connection header received
0927 02:36:10.165796 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:10.165802 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:10.165946 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:10.166054 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:36:10.166702 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:10.167452 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:10.168399 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:10.168864 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:10.169023 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:10.171361 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:36:10.171369 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:36:10.171380 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:36:10.171405 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:36:10.172999 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:10.173474 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:10.173478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:10.173479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:10.173549 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:10.173795 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:10.173798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:10.173799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:10.174018 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:36:10.174126 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:10.174524 (+ 398us) server_negotiation.cc:299] Negotiation successful
0927 02:36:10.174666 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":305,"thread_start_us":120,"threads_started":1}
I20260927 02:36:10.175758 14790 tablet_service.cc:1514] Processing DeleteTablet for tablet fa839b6e084842b78ee07bfd820649a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:36:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50138
I20260927 02:36:10.176556 14874 tablet_replica.cc:330] stopping tablet replica
I20260927 02:36:10.176836 14874 raft_consensus.cc:2229] T fa839b6e084842b78ee07bfd820649a3 P d978229ba60143278b1226ab8132033d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:36:10.177093 14874 raft_consensus.cc:2258] T fa839b6e084842b78ee07bfd820649a3 P d978229ba60143278b1226ab8132033d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:36:10.178054 14874 ts_tablet_manager.cc:1905] T fa839b6e084842b78ee07bfd820649a3 P d978229ba60143278b1226ab8132033d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:36:10.181437 14874 ts_tablet_manager.cc:1918] T fa839b6e084842b78ee07bfd820649a3 P d978229ba60143278b1226ab8132033d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:36:10.181524 14874 log.cc:1196] T fa839b6e084842b78ee07bfd820649a3 P d978229ba60143278b1226ab8132033d: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-0/wal/wals/fa839b6e084842b78ee07bfd820649a3
I20260927 02:36:10.181732 14874 ts_tablet_manager.cc:1939] T fa839b6e084842b78ee07bfd820649a3 P d978229ba60143278b1226ab8132033d: Deleting consensus metadata
I20260927 02:36:10.183266 14661 catalog_manager.cc:4841] TS d978229ba60143278b1226ab8132033d (127.0.98.1:45151): tablet fa839b6e084842b78ee07bfd820649a3 (table test-table [id=3f42e309bd844ee8bdb0484d82b55f9f]) successfully deleted
W20260927 02:36:10.254345 14869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:36:10.254598 14869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:36:10.254654 14869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:36:10.258548 14869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:36:10.258605 14869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:36:10.258682 14869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:36:10.263652 14869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34453
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=40601
--tserver_master_addrs=127.0.98.62:36259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:36:10.264578 14869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:36:10.265520 14869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:36:10.274719 14879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:36:10.274775 14880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:36:10.274750 14882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:36:10.275017 14869 server_base.cc:1034] running on GCE node
I20260927 02:36:10.275463 14869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:36:10.276099 14869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:36:10.277428 14869 hybrid_clock.cc:648] HybridClock initialized: now 1790476570277373 us; error 63 us; skew 500 ppm
Sep 27 02:36:10 dist-test-slave-grlk krb5kdc[14097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:36:10.280236 14869 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:36:10.281162 14869 webserver.cc:460] Webserver started at http://127.0.98.2:40601/ using document root <none> and password file <none>
I20260927 02:36:10.281687 14869 fs_manager.cc:359] Metadata directory not provided
I20260927 02:36:10.281762 14869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:36:10.285590 14869 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:36:10.287611 14889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:10.288760 14869 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:36:10.288865 14869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-1/wal
uuid: "64a55ced77b1438e87fbb5e5feb28738"
format_stamp: "Formatted at 2026-09-27 02:36:09 on dist-test-slave-grlk"
server_key: "7bb7680e522f3ba3c3b126c4092962a7"
server_key_iv: "c2d0b22f8f814c8396a18878bcbd04dd"
server_key_version: "encryptionkey@0"
I20260927 02:36:10.289443 14869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:36:10.309333 14869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:36:10.311874 14869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:36:10.312043 14869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:36:10.312587 14869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:36:10.313372 14869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:36:10.313418 14869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:10.313469 14869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:36:10.313498 14869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:10.321578 14869 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34453
I20260927 02:36:10.321614 15002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34453 every 8 connection(s)
I20260927 02:36:10.322469 14869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:36:10.326047 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 14869
I20260927 02:36:10.326498 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39691
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36259
--webserver_port=42463
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 02:36:10 dist-test-slave-grlk krb5kdc[14097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:36:10.337435 14867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:10.324067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41853 (local address 127.0.98.62:36259)
0927 02:36:10.324184 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:10.324187 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:10.324917 (+ 730us) server_negotiation.cc:407] Connection header received
0927 02:36:10.326299 (+ 1382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:10.326302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:10.326353 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:10.326444 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:36:10.327967 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:10.328763 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:10.329890 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:10.330248 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:10.330361 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:10.332836 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:36:10.332839 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:36:10.332841 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:36:10.332870 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:36:10.334694 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:10.335466 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:10.335470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:10.335472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:10.335597 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:10.336090 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:10.336094 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:10.336096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:10.336403 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:36:10.336488 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:10.337155 (+ 667us) server_negotiation.cc:299] Negotiation successful
0927 02:36:10.337274 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 02:36:10.338461 15005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:10.324384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.2:41853)
0927 02:36:10.324776 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:10.324815 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:10.326051 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:10.326661 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:36:10.326670 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:36:10.327101 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:36:10.327748 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:10.327768 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:10.328883 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:10.328886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:10.329771 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:10.329780 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:10.330384 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:10.330387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:10.330538 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:10.331093 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:36:10.331108 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:36:10.332606 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:36:10.334831 (+ 2225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:10.334838 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:10.334858 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:10.335265 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:10.335724 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:10.335728 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:10.335731 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:10.335974 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:10.336495 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:36:10.336502 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:36:10.336856 (+ 354us) client_negotiation.cc:769] Sending connection context
0927 02:36:10.337123 (+ 267us) client_negotiation.cc:240] Negotiation successful
0927 02:36:10.337462 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":98,"threads_started":1}
I20260927 02:36:10.339627 15003 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36259
I20260927 02:36:10.339855 15003 heartbeater.cc:464] Registering TS with master...
I20260927 02:36:10.340353 15003 heartbeater.cc:511] Master 127.0.98.62:36259 requested a full tablet report, sending...
I20260927 02:36:10.341339 14677 ts_manager.cc:194] Registered new tserver with Master: 64a55ced77b1438e87fbb5e5feb28738 (127.0.98.2:34453)
I20260927 02:36:10.341895 14677 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41853
W20260927 02:36:10.432987 15006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:36:10.433205 15006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:36:10.433248 15006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:36:10.437150 15006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:36:10.437201 15006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:36:10.437268 15006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:36:10.442245 15006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39691
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42463
--tserver_master_addrs=127.0.98.62:36259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:36:10.443182 15006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:36:10.444087 15006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:36:10.452975 15014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:36:10.453079 15006 server_base.cc:1034] running on GCE node
W20260927 02:36:10.452984 15012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:36:10.452994 15011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:36:10.453600 15006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:36:10.454217 15006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:36:10.455364 15006 hybrid_clock.cc:648] HybridClock initialized: now 1790476570455339 us; error 40 us; skew 500 ppm
Sep 27 02:36:10 dist-test-slave-grlk krb5kdc[14097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:36:10.457845 15006 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:36:10.458613 15006 webserver.cc:460] Webserver started at http://127.0.98.3:42463/ using document root <none> and password file <none>
I20260927 02:36:10.459111 15006 fs_manager.cc:359] Metadata directory not provided
I20260927 02:36:10.459185 15006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:36:10.462834 15006 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260927 02:36:10.464649 15021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:10.465768 15006 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260927 02:36:10.465867 15006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-2/wal
uuid: "0b950ed19247482b9e0ff9b7a7d9eac1"
format_stamp: "Formatted at 2026-09-27 02:36:09 on dist-test-slave-grlk"
server_key: "96a4282c3ae44acfb5d64bd5a83d3b57"
server_key_iv: "bcddd3a4e526610621bd2bb38e779a47"
server_key_version: "encryptionkey@0"
I20260927 02:36:10.466367 15006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:36:10.481897 15006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:36:10.484407 15006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:36:10.484560 15006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:36:10.485172 15006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:36:10.485903 15006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:36:10.485948 15006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:10.486011 15006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:36:10.486059 15006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:10.494124 15006 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39691
I20260927 02:36:10.494168 15134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39691 every 8 connection(s)
I20260927 02:36:10.495033 15006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:36:10.502815 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 15006
Sep 27 02:36:10 dist-test-slave-grlk krb5kdc[14097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:36:10.508095 14867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:10.496559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58177 (local address 127.0.98.62:36259)
0927 02:36:10.496704 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:10.496708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:10.497521 (+ 813us) server_negotiation.cc:407] Connection header received
0927 02:36:10.498692 (+ 1171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:10.498695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:10.498739 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:10.498809 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:36:10.499927 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:10.500522 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:10.501891 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:10.502129 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:10.502222 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:10.504674 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:36:10.504678 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:36:10.504681 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:36:10.504706 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:36:10.506151 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:10.506683 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:10.506686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:10.506687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:10.506761 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:10.507079 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:10.507082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:10.507084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:10.507268 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:36:10.507365 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:10.507818 (+ 453us) server_negotiation.cc:299] Negotiation successful
0927 02:36:10.507927 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 02:36:10.509032 15137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:10.496962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.3:58177)
0927 02:36:10.497386 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:10.497424 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:10.498503 (+ 1079us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:10.498950 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:36:10.498956 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:36:10.499292 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:36:10.499781 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:10.499801 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:10.500747 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:10.500750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:10.501709 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:10.501718 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:10.502258 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:10.502260 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:10.502396 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:10.502939 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:36:10.502957 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:36:10.504514 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:36:10.506254 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:10.506259 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:10.506270 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:10.506588 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:10.506843 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:10.506846 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:10.506848 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:10.506990 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:10.507361 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:36:10.507366 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:36:10.507598 (+ 232us) client_negotiation.cc:769] Sending connection context
0927 02:36:10.507749 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 02:36:10.508003 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":106,"threads_started":1}
I20260927 02:36:10.510185 15135 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36259
I20260927 02:36:10.510428 15135 heartbeater.cc:464] Registering TS with master...
I20260927 02:36:10.510979 15135 heartbeater.cc:511] Master 127.0.98.62:36259 requested a full tablet report, sending...
I20260927 02:36:10.511344 15139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:10.503468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55146 (local address 127.0.98.62:36259)
0927 02:36:10.503731 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:10.503735 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:10.503847 (+ 112us) server_negotiation.cc:407] Connection header received
0927 02:36:10.503935 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:10.503938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:10.503985 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:10.504075 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:36:10.504731 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:10.505269 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:10.506399 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:10.506735 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:10.506810 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:10.508172 (+ 1362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:36:10.508175 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:36:10.508178 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:36:10.508201 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:36:10.509836 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:10.510257 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:10.510260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:10.510262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:10.510347 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:10.510596 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:10.510599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:10.510601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:10.510800 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:36:10.510997 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:10.511089 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 02:36:10.511197 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":102,"threads_started":1}
I20260927 02:36:10.511987 14677 ts_manager.cc:194] Registered new tserver with Master: 0b950ed19247482b9e0ff9b7a7d9eac1 (127.0.98.3:39691)
I20260927 02:36:10.512560 14677 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58177
I20260927 02:36:10.513052 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:36:10.516901 15139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:10.513599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55152 (local address 127.0.98.62:36259)
0927 02:36:10.513695 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:10.513699 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:10.513867 (+ 168us) server_negotiation.cc:407] Connection header received
0927 02:36:10.513993 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:10.513996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:10.514089 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:10.514178 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:36:10.514386 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:10.514865 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:10.515665 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:10.515905 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:10.515986 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:10.516261 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:36:10.516578 (+ 317us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:36:10.516671 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:10.516739 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 02:36:10.516792 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 02:36:10.517418 14677 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:55152:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:36:10.520546 14677 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:36:10.527710 14790 tablet_service.cc:1467] Processing CreateTablet for tablet 9b75852427204cc5a1a24b8e68734fce (DEFAULT_TABLE table=test-table [id=74f83225480a4ef890102dd551fbc7ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:36:10.528318 14790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b75852427204cc5a1a24b8e68734fce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:36:10.530854 14757 tablet_bootstrap.cc:492] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d: Bootstrap starting.
I20260927 02:36:10.531627 14872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:10.526575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39691 (local address 127.0.0.1:43690)
0927 02:36:10.526721 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:10.526734 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:10.526803 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:10.527449 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:36:10.527452 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:36:10.527470 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:36:10.527560 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:10.527566 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:10.529036 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:10.529040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:10.530273 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:10.530282 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:10.531223 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:10.531226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:10.531326 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:10.531366 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 02:36:10.531433 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 02:36:10.531509 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260927 02:36:10.531669 14757 tablet_bootstrap.cc:654] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d: Neither blocks nor log segments found. Creating new log.
I20260927 02:36:10.532085 15146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:10.527107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46130 (local address 127.0.98.2:34453)
0927 02:36:10.527395 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:10.527405 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:10.527551 (+ 146us) server_negotiation.cc:407] Connection header received
0927 02:36:10.527670 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:10.527675 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:10.527813 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:10.527942 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:36:10.528228 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:10.529070 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:10.530523 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:10.531258 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:10.531354 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:10.531475 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:10.531648 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 02:36:10.531936 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":95,"threads_started":1}
I20260927 02:36:10.532107 15144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:10.526705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43690 (local address 127.0.98.3:39691)
0927 02:36:10.527001 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:10.527011 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:10.527032 (+ 21us) server_negotiation.cc:407] Connection header received
0927 02:36:10.527089 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:10.527093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:10.527275 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:10.527702 (+ 427us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:36:10.527858 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:10.528866 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:10.530397 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:10.531119 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:10.531209 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:10.531341 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:10.531877 (+ 536us) server_negotiation.cc:299] Negotiation successful
0927 02:36:10.531980 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":127,"threads_started":1}
I20260927 02:36:10.532455 15145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:10.526854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34453 (local address 127.0.0.1:46130)
0927 02:36:10.527466 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:10.527478 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:10.527571 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:10.528021 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:36:10.528024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:36:10.528040 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:36:10.528132 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:10.528137 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:10.529330 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:10.529333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:10.530423 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:10.530430 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:10.531425 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:10.531428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:10.531525 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:10.531544 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 02:36:10.531608 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 02:36:10.532327 (+ 719us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":87,"threads_started":1}
I20260927 02:36:10.533298 14757 tablet_bootstrap.cc:492] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d: No bootstrap required, opened a new log
I20260927 02:36:10.533380 14757 ts_tablet_manager.cc:1397] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 02:36:10.533738 14757 raft_consensus.cc:348] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b950ed19247482b9e0ff9b7a7d9eac1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39691 } } peers { permanent_uuid: "64a55ced77b1438e87fbb5e5feb28738" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34453 } } peers { permanent_uuid: "d978229ba60143278b1226ab8132033d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45151 } }
I20260927 02:36:10.533828 14757 raft_consensus.cc:374] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:36:10.533864 14757 raft_consensus.cc:729] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d978229ba60143278b1226ab8132033d, State: Initialized, Role: FOLLOWER
I20260927 02:36:10.533977 14757 consensus_queue.cc:260] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b950ed19247482b9e0ff9b7a7d9eac1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39691 } } peers { permanent_uuid: "64a55ced77b1438e87fbb5e5feb28738" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34453 } } peers { permanent_uuid: "d978229ba60143278b1226ab8132033d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45151 } }
I20260927 02:36:10.534235 14757 ts_tablet_manager.cc:1428] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 02:36:10.534819 15069 tablet_service.cc:1467] Processing CreateTablet for tablet 9b75852427204cc5a1a24b8e68734fce (DEFAULT_TABLE table=test-table [id=74f83225480a4ef890102dd551fbc7ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:36:10.535005 14937 tablet_service.cc:1467] Processing CreateTablet for tablet 9b75852427204cc5a1a24b8e68734fce (DEFAULT_TABLE table=test-table [id=74f83225480a4ef890102dd551fbc7ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:36:10.535868 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b75852427204cc5a1a24b8e68734fce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:36:10.535876 15069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b75852427204cc5a1a24b8e68734fce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:36:10.543452 15149 tablet_bootstrap.cc:492] T 9b75852427204cc5a1a24b8e68734fce P 0b950ed19247482b9e0ff9b7a7d9eac1: Bootstrap starting.
I20260927 02:36:10.544090 15148 tablet_bootstrap.cc:492] T 9b75852427204cc5a1a24b8e68734fce P 64a55ced77b1438e87fbb5e5feb28738: Bootstrap starting.
I20260927 02:36:10.545782 15149 tablet_bootstrap.cc:654] T 9b75852427204cc5a1a24b8e68734fce P 0b950ed19247482b9e0ff9b7a7d9eac1: Neither blocks nor log segments found. Creating new log.
I20260927 02:36:10.546448 15148 tablet_bootstrap.cc:654] T 9b75852427204cc5a1a24b8e68734fce P 64a55ced77b1438e87fbb5e5feb28738: Neither blocks nor log segments found. Creating new log.
I20260927 02:36:10.546970 15149 log.cc:824] T 9b75852427204cc5a1a24b8e68734fce P 0b950ed19247482b9e0ff9b7a7d9eac1: Log is configured to *not* fsync() on all Append() calls
I20260927 02:36:10.547160 15148 log.cc:824] T 9b75852427204cc5a1a24b8e68734fce P 64a55ced77b1438e87fbb5e5feb28738: Log is configured to *not* fsync() on all Append() calls
I20260927 02:36:10.548605 15149 tablet_bootstrap.cc:492] T 9b75852427204cc5a1a24b8e68734fce P 0b950ed19247482b9e0ff9b7a7d9eac1: No bootstrap required, opened a new log
I20260927 02:36:10.548663 15148 tablet_bootstrap.cc:492] T 9b75852427204cc5a1a24b8e68734fce P 64a55ced77b1438e87fbb5e5feb28738: No bootstrap required, opened a new log
I20260927 02:36:10.548779 15149 ts_tablet_manager.cc:1397] T 9b75852427204cc5a1a24b8e68734fce P 0b950ed19247482b9e0ff9b7a7d9eac1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 02:36:10.548825 15148 ts_tablet_manager.cc:1397] T 9b75852427204cc5a1a24b8e68734fce P 64a55ced77b1438e87fbb5e5feb28738: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 02:36:10.551594 15148 raft_consensus.cc:348] T 9b75852427204cc5a1a24b8e68734fce P 64a55ced77b1438e87fbb5e5feb28738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b950ed19247482b9e0ff9b7a7d9eac1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39691 } } peers { permanent_uuid: "64a55ced77b1438e87fbb5e5feb28738" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34453 } } peers { permanent_uuid: "d978229ba60143278b1226ab8132033d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45151 } }
I20260927 02:36:10.551596 15149 raft_consensus.cc:348] T 9b75852427204cc5a1a24b8e68734fce P 0b950ed19247482b9e0ff9b7a7d9eac1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b950ed19247482b9e0ff9b7a7d9eac1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39691 } } peers { permanent_uuid: "64a55ced77b1438e87fbb5e5feb28738" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34453 } } peers { permanent_uuid: "d978229ba60143278b1226ab8132033d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45151 } }
I20260927 02:36:10.551795 15148 raft_consensus.cc:374] T 9b75852427204cc5a1a24b8e68734fce P 64a55ced77b1438e87fbb5e5feb28738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:36:10.551797 15149 raft_consensus.cc:374] T 9b75852427204cc5a1a24b8e68734fce P 0b950ed19247482b9e0ff9b7a7d9eac1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:36:10.551833 15148 raft_consensus.cc:729] T 9b75852427204cc5a1a24b8e68734fce P 64a55ced77b1438e87fbb5e5feb28738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64a55ced77b1438e87fbb5e5feb28738, State: Initialized, Role: FOLLOWER
I20260927 02:36:10.551854 15149 raft_consensus.cc:729] T 9b75852427204cc5a1a24b8e68734fce P 0b950ed19247482b9e0ff9b7a7d9eac1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b950ed19247482b9e0ff9b7a7d9eac1, State: Initialized, Role: FOLLOWER
I20260927 02:36:10.552197 15148 consensus_queue.cc:260] T 9b75852427204cc5a1a24b8e68734fce P 64a55ced77b1438e87fbb5e5feb28738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b950ed19247482b9e0ff9b7a7d9eac1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39691 } } peers { permanent_uuid: "64a55ced77b1438e87fbb5e5feb28738" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34453 } } peers { permanent_uuid: "d978229ba60143278b1226ab8132033d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45151 } }
I20260927 02:36:10.552213 15149 consensus_queue.cc:260] T 9b75852427204cc5a1a24b8e68734fce P 0b950ed19247482b9e0ff9b7a7d9eac1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b950ed19247482b9e0ff9b7a7d9eac1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39691 } } peers { permanent_uuid: "64a55ced77b1438e87fbb5e5feb28738" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34453 } } peers { permanent_uuid: "d978229ba60143278b1226ab8132033d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45151 } }
I20260927 02:36:10.552735 15003 heartbeater.cc:503] Master 127.0.98.62:36259 was elected leader, sending a full tablet report...
I20260927 02:36:10.553041 15149 ts_tablet_manager.cc:1428] T 9b75852427204cc5a1a24b8e68734fce P 0b950ed19247482b9e0ff9b7a7d9eac1: Time spent starting tablet: real 0.004s user 0.005s sys 0.001s
I20260927 02:36:10.553057 15148 ts_tablet_manager.cc:1428] T 9b75852427204cc5a1a24b8e68734fce P 64a55ced77b1438e87fbb5e5feb28738: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260927 02:36:10.553290 15135 heartbeater.cc:503] Master 127.0.98.62:36259 was elected leader, sending a full tablet report...
W20260927 02:36:10.573246 15004 tablet.cc:2369] T 9b75852427204cc5a1a24b8e68734fce P 64a55ced77b1438e87fbb5e5feb28738: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:36:10.637377 14866 tablet.cc:2369] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:36:10.745935 15136 tablet.cc:2369] T 9b75852427204cc5a1a24b8e68734fce P 0b950ed19247482b9e0ff9b7a7d9eac1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:36:10.804387 14871 raft_consensus.cc:482] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:36:10.804644 14871 raft_consensus.cc:504] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b950ed19247482b9e0ff9b7a7d9eac1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39691 } } peers { permanent_uuid: "64a55ced77b1438e87fbb5e5feb28738" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34453 } } peers { permanent_uuid: "d978229ba60143278b1226ab8132033d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45151 } }
I20260927 02:36:10.805898 14871 leader_election.cc:288] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b950ed19247482b9e0ff9b7a7d9eac1 (127.0.98.3:39691), 64a55ced77b1438e87fbb5e5feb28738 (127.0.98.2:34453)
I20260927 02:36:10.809830 15146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:10.806251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49059 (local address 127.0.98.2:34453)
0927 02:36:10.806403 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:10.806409 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:10.806782 (+ 373us) server_negotiation.cc:407] Connection header received
0927 02:36:10.806913 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:10.806916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:10.806966 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:10.807045 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:36:10.807296 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:10.807819 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:10.808854 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:10.809262 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:10.809338 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:10.809393 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:10.809644 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 02:36:10.809710 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 02:36:10.809852 15154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:10.806141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39691 (local address 127.0.98.1:58971)
0927 02:36:10.806437 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:10.806454 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:10.806581 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:10.806888 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:36:10.806890 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:36:10.806910 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:36:10.807047 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:10.807054 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:10.807826 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:10.807829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:10.808716 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:10.808724 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:10.809435 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:10.809438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:10.809510 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:10.809541 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 02:36:10.809606 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 02:36:10.809680 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":57,"thread_start_us":106,"threads_started":1}
I20260927 02:36:10.809898 15155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:10.806141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34453 (local address 127.0.98.1:49059)
0927 02:36:10.806666 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:10.806682 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:10.806805 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:10.807065 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:36:10.807067 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:36:10.807081 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:36:10.807181 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:10.807188 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:10.807926 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:10.807929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:10.808731 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:10.808740 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:10.809384 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:10.809389 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:10.809477 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:10.809535 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 02:36:10.809619 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 02:36:10.809704 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"mutex_wait_us":245,"thread_start_us":79,"threads_started":1}
I20260927 02:36:10.809837 15144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:10.806261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58971 (local address 127.0.98.3:39691)
0927 02:36:10.806405 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:10.806409 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:10.806537 (+ 128us) server_negotiation.cc:407] Connection header received
0927 02:36:10.806713 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:10.806717 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:10.806779 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:10.806881 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:36:10.807153 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:10.807711 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:10.808841 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:10.809306 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:10.809419 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:10.809486 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:10.809635 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 02:36:10.809704 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 02:36:10.810531 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b75852427204cc5a1a24b8e68734fce" candidate_uuid: "d978229ba60143278b1226ab8132033d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64a55ced77b1438e87fbb5e5feb28738" is_pre_election: true
I20260927 02:36:10.810556 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b75852427204cc5a1a24b8e68734fce" candidate_uuid: "d978229ba60143278b1226ab8132033d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b950ed19247482b9e0ff9b7a7d9eac1" is_pre_election: true
I20260927 02:36:10.810811 14957 raft_consensus.cc:2454] T 9b75852427204cc5a1a24b8e68734fce P 64a55ced77b1438e87fbb5e5feb28738 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d978229ba60143278b1226ab8132033d in term 0.
I20260927 02:36:10.810810 15089 raft_consensus.cc:2454] T 9b75852427204cc5a1a24b8e68734fce P 0b950ed19247482b9e0ff9b7a7d9eac1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d978229ba60143278b1226ab8132033d in term 0.
I20260927 02:36:10.811208 14751 leader_election.cc:302] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64a55ced77b1438e87fbb5e5feb28738, d978229ba60143278b1226ab8132033d; no voters:
I20260927 02:36:10.811455 14871 raft_consensus.cc:2781] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:36:10.811534 14871 raft_consensus.cc:482] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:36:10.811569 14871 raft_consensus.cc:3037] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:36:10.812534 14871 raft_consensus.cc:504] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b950ed19247482b9e0ff9b7a7d9eac1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39691 } } peers { permanent_uuid: "64a55ced77b1438e87fbb5e5feb28738" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34453 } } peers { permanent_uuid: "d978229ba60143278b1226ab8132033d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45151 } }
I20260927 02:36:10.812872 14871 leader_election.cc:288] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d [CANDIDATE]: Term 1 election: Requested vote from peers 0b950ed19247482b9e0ff9b7a7d9eac1 (127.0.98.3:39691), 64a55ced77b1438e87fbb5e5feb28738 (127.0.98.2:34453)
I20260927 02:36:10.813194 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b75852427204cc5a1a24b8e68734fce" candidate_uuid: "d978229ba60143278b1226ab8132033d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b950ed19247482b9e0ff9b7a7d9eac1"
I20260927 02:36:10.813230 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b75852427204cc5a1a24b8e68734fce" candidate_uuid: "d978229ba60143278b1226ab8132033d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64a55ced77b1438e87fbb5e5feb28738"
I20260927 02:36:10.813305 15089 raft_consensus.cc:3037] T 9b75852427204cc5a1a24b8e68734fce P 0b950ed19247482b9e0ff9b7a7d9eac1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:36:10.813338 14957 raft_consensus.cc:3037] T 9b75852427204cc5a1a24b8e68734fce P 64a55ced77b1438e87fbb5e5feb28738 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:36:10.814271 15089 raft_consensus.cc:2454] T 9b75852427204cc5a1a24b8e68734fce P 0b950ed19247482b9e0ff9b7a7d9eac1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d978229ba60143278b1226ab8132033d in term 1.
I20260927 02:36:10.814327 14957 raft_consensus.cc:2454] T 9b75852427204cc5a1a24b8e68734fce P 64a55ced77b1438e87fbb5e5feb28738 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d978229ba60143278b1226ab8132033d in term 1.
I20260927 02:36:10.814642 14751 leader_election.cc:302] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64a55ced77b1438e87fbb5e5feb28738, d978229ba60143278b1226ab8132033d; no voters:
I20260927 02:36:10.814829 14871 raft_consensus.cc:2781] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:36:10.815006 14871 raft_consensus.cc:686] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d [term 1 LEADER]: Becoming Leader. State: Replica: d978229ba60143278b1226ab8132033d, State: Running, Role: LEADER
I20260927 02:36:10.815248 14871 consensus_queue.cc:237] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b950ed19247482b9e0ff9b7a7d9eac1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39691 } } peers { permanent_uuid: "64a55ced77b1438e87fbb5e5feb28738" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34453 } } peers { permanent_uuid: "d978229ba60143278b1226ab8132033d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45151 } }
I20260927 02:36:10.818069 14676 catalog_manager.cc:5495] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d reported cstate change: term changed from 0 to 1, leader changed from <none> to d978229ba60143278b1226ab8132033d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "d978229ba60143278b1226ab8132033d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b950ed19247482b9e0ff9b7a7d9eac1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64a55ced77b1438e87fbb5e5feb28738" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d978229ba60143278b1226ab8132033d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45151 } health_report { overall_health: HEALTHY } } }
I20260927 02:36:10.895174 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:10.891796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57966 (local address 127.0.98.1:45151)
0927 02:36:10.892049 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:10.892056 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:10.892069 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:36:10.892140 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:10.892142 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:10.892194 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:10.892289 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:36:10.892468 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:10.893075 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:10.893923 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:10.894177 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:10.894254 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:10.894432 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:36:10.894848 (+ 416us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:36:10.894929 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:10.894973 (+ 44us) server_negotiation.cc:299] Negotiation successful
0927 02:36:10.895020 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":96,"threads_started":1}
I20260927 02:36:10.898644 14957 raft_consensus.cc:1261] T 9b75852427204cc5a1a24b8e68734fce P 64a55ced77b1438e87fbb5e5feb28738 [term 1 FOLLOWER]: Refusing update from remote peer d978229ba60143278b1226ab8132033d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:36:10.898644 15089 raft_consensus.cc:1261] T 9b75852427204cc5a1a24b8e68734fce P 0b950ed19247482b9e0ff9b7a7d9eac1 [term 1 FOLLOWER]: Refusing update from remote peer d978229ba60143278b1226ab8132033d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:36:10.899302 15156 consensus_queue.cc:1035] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b950ed19247482b9e0ff9b7a7d9eac1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:36:10.899513 14871 consensus_queue.cc:1035] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d [LEADER]: Connected to new peer: Peer: permanent_uuid: "64a55ced77b1438e87fbb5e5feb28738" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:36:10.905896 15160 mvcc.cc:205] Tried to move back new op lower bound from 7333792034395279360 to 7333792034061828096. Current Snapshot: MvccSnapshot[applied={T|T < 7333792034395279360}]
I20260927 02:36:10.913738 14676 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:55152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:36:10.913906 14676 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:55152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:36:10.916142 14676 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e: Sending DeleteTablet for 3 replicas of tablet 9b75852427204cc5a1a24b8e68734fce
I20260927 02:36:10.916885 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 14715
I20260927 02:36:10.916904 15069 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b75852427204cc5a1a24b8e68734fce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:36:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43690
I20260927 02:36:10.916908 14937 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b75852427204cc5a1a24b8e68734fce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:36:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46130
I20260927 02:36:10.918378 15170 tablet_replica.cc:330] T 9b75852427204cc5a1a24b8e68734fce P 64a55ced77b1438e87fbb5e5feb28738: stopping tablet replica
I20260927 02:36:10.918637 15170 raft_consensus.cc:2229] T 9b75852427204cc5a1a24b8e68734fce P 64a55ced77b1438e87fbb5e5feb28738 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:36:10.918895 15170 raft_consensus.cc:2258] T 9b75852427204cc5a1a24b8e68734fce P 64a55ced77b1438e87fbb5e5feb28738 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:36:10.919194 15171 tablet_replica.cc:330] T 9b75852427204cc5a1a24b8e68734fce P 0b950ed19247482b9e0ff9b7a7d9eac1: stopping tablet replica
I20260927 02:36:10.919412 15171 raft_consensus.cc:2229] T 9b75852427204cc5a1a24b8e68734fce P 0b950ed19247482b9e0ff9b7a7d9eac1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:36:10.919628 15171 raft_consensus.cc:2258] T 9b75852427204cc5a1a24b8e68734fce P 0b950ed19247482b9e0ff9b7a7d9eac1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:36:10.920879 15171 ts_tablet_manager.cc:1905] T 9b75852427204cc5a1a24b8e68734fce P 0b950ed19247482b9e0ff9b7a7d9eac1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:36:10.921969 15170 ts_tablet_manager.cc:1905] T 9b75852427204cc5a1a24b8e68734fce P 64a55ced77b1438e87fbb5e5feb28738: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 02:36:10.923328 14661 connection.cc:535] client connection to 127.0.98.1:45151 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 02:36:10.923475 14661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 02:36:10.923591 15171 ts_tablet_manager.cc:1918] T 9b75852427204cc5a1a24b8e68734fce P 0b950ed19247482b9e0ff9b7a7d9eac1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:36:10.923681 15171 log.cc:1196] T 9b75852427204cc5a1a24b8e68734fce P 0b950ed19247482b9e0ff9b7a7d9eac1: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-2/wal/wals/9b75852427204cc5a1a24b8e68734fce
I20260927 02:36:10.924017 15171 ts_tablet_manager.cc:1939] T 9b75852427204cc5a1a24b8e68734fce P 0b950ed19247482b9e0ff9b7a7d9eac1: Deleting consensus metadata
I20260927 02:36:10.924700 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 14869
I20260927 02:36:10.924711 15145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:10.924397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45151 (local address 127.0.0.1:57982)
0927 02:36:10.924540 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:10.924616 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 02:36:10.924768 15170 ts_tablet_manager.cc:1918] T 9b75852427204cc5a1a24b8e68734fce P 64a55ced77b1438e87fbb5e5feb28738: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:36:10.924954 15170 log.cc:1196] T 9b75852427204cc5a1a24b8e68734fce P 64a55ced77b1438e87fbb5e5feb28738: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-1/wal/wals/9b75852427204cc5a1a24b8e68734fce
I20260927 02:36:10.925349 14662 catalog_manager.cc:4841] TS 0b950ed19247482b9e0ff9b7a7d9eac1 (127.0.98.3:39691): tablet 9b75852427204cc5a1a24b8e68734fce (table test-table [id=74f83225480a4ef890102dd551fbc7ad]) successfully deleted
W20260927 02:36:10.930804 14661 catalog_manager.cc:4584] TS d978229ba60143278b1226ab8132033d (127.0.98.1:45151): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9b75852427204cc5a1a24b8e68734fce: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45151: connect: Connection refused (error 111)
I20260927 02:36:10.931450 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 15006
I20260927 02:36:10.933393 15145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:10.933137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34453 (local address 127.0.0.1:48060)
0927 02:36:10.933264 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:10.933320 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:34453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 02:36:10.936471 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 14644
I20260927 02:36:10.941682 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36259
--webserver_interface=127.0.98.62
--webserver_port=42597
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36259
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:36:11.046372 15173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:36:11.046641 15173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:36:11.046701 15173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:36:11.051170 15173 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:36:11.051234 15173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:36:11.051258 15173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:36:11.051281 15173 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:36:11.051309 15173 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:36:11.056993 15173 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:36:11.058040 15173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:36:11.059016 15173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:36:11.067037 15178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:36:11.067147 15181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:36:11.067093 15179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:36:11.067310 15173 server_base.cc:1034] running on GCE node
I20260927 02:36:11.067814 15173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:36:11.068851 15173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:36:11.070009 15173 hybrid_clock.cc:648] HybridClock initialized: now 1790476571069992 us; error 28 us; skew 500 ppm
Sep 27 02:36:11 dist-test-slave-grlk krb5kdc[14097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:36:11.072541 15173 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:36:11.073350 15173 webserver.cc:460] Webserver started at http://127.0.98.62:42597/ using document root <none> and password file <none>
I20260927 02:36:11.073913 15173 fs_manager.cc:359] Metadata directory not provided
I20260927 02:36:11.073979 15173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:36:11.078225 15173 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:36:11.080336 15188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:11.081619 15173 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:36:11.081732 15173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "b53ecc8c654d41c7a4e117f1ef1d345e"
format_stamp: "Formatted at 2026-09-27 02:36:08 on dist-test-slave-grlk"
server_key: "c22e67050e2607299641684c3b867e42"
server_key_iv: "eefafd85429342ad55b0acd8de8f4e9a"
server_key_version: "encryptionkey@0"
I20260927 02:36:11.082263 15173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:36:11.093710 15173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:36:11.096577 15173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:36:11.096745 15173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:36:11.102900 15173 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36259
I20260927 02:36:11.103135 15240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36259 every 8 connection(s)
I20260927 02:36:11.103868 15173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:36:11.107064 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 15173
I20260927 02:36:11.107434 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45151
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36259
--webserver_port=43069
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:36:11.107913 15241 sys_catalog.cc:263] Verifying existing consensus state
I20260927 02:36:11.110129 15241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e: Bootstrap starting.
I20260927 02:36:11.128103 15241 log.cc:824] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e: Log is configured to *not* fsync() on all Append() calls
I20260927 02:36:11.138945 15241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:36:11.139284 15241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e: Bootstrap complete.
I20260927 02:36:11.142959 15241 raft_consensus.cc:348] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53ecc8c654d41c7a4e117f1ef1d345e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36259 } }
I20260927 02:36:11.143318 15241 raft_consensus.cc:729] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b53ecc8c654d41c7a4e117f1ef1d345e, State: Initialized, Role: FOLLOWER
I20260927 02:36:11.143774 15241 consensus_queue.cc:260] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53ecc8c654d41c7a4e117f1ef1d345e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36259 } }
I20260927 02:36:11.143890 15241 raft_consensus.cc:388] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:36:11.143951 15241 raft_consensus.cc:482] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:36:11.144035 15241 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [term 2 FOLLOWER]: Advancing to term 3
I20260927 02:36:11.145835 15241 raft_consensus.cc:504] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53ecc8c654d41c7a4e117f1ef1d345e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36259 } }
I20260927 02:36:11.146171 15241 leader_election.cc:302] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b53ecc8c654d41c7a4e117f1ef1d345e; no voters:
I20260927 02:36:11.146449 15241 leader_election.cc:288] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 02:36:11.146574 15245 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [term 3 FOLLOWER]: Leader election won for term 3
I20260927 02:36:11.146760 15245 raft_consensus.cc:686] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [term 3 LEADER]: Becoming Leader. State: Replica: b53ecc8c654d41c7a4e117f1ef1d345e, State: Running, Role: LEADER
I20260927 02:36:11.147032 15245 consensus_queue.cc:237] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53ecc8c654d41c7a4e117f1ef1d345e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36259 } }
I20260927 02:36:11.147603 15241 sys_catalog.cc:564] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:36:11.148241 15246 sys_catalog.cc:455] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "b53ecc8c654d41c7a4e117f1ef1d345e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53ecc8c654d41c7a4e117f1ef1d345e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36259 } } }
I20260927 02:36:11.148365 15246 sys_catalog.cc:458] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [sys.catalog]: This master's current role is: LEADER
I20260927 02:36:11.148653 15247 sys_catalog.cc:455] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b53ecc8c654d41c7a4e117f1ef1d345e. Latest consensus state: current_term: 3 leader_uuid: "b53ecc8c654d41c7a4e117f1ef1d345e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53ecc8c654d41c7a4e117f1ef1d345e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36259 } } }
I20260927 02:36:11.148734 15247 sys_catalog.cc:458] T 00000000000000000000000000000000 P b53ecc8c654d41c7a4e117f1ef1d345e [sys.catalog]: This master's current role is: LEADER
I20260927 02:36:11.149250 15252 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:36:11.154197 15252 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:36:11.154959 15252 catalog_manager.cc:1259] Loaded cluster ID: f86887c5d3c2418f83ba2792ec8f45bf
I20260927 02:36:11.155026 15252 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 02:36:11.155951 15252 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f4146def9ad google::LogMessage::Fail() at ??:0
@ 0x7f4146df288a google::LogMessage::SendToLog() at ??:0
@ 0x7f4146def4de google::LogMessage::Flush() at ??:0
@ 0x7f4146df0a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f415172efdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f415172fd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f415172c579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f4151770640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f41490413d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f4149041c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f41490433f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f4149032512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f414a6766db start_thread at ??:0
@ 0x7f41458ed61f clone at ??:0
W20260927 02:36:11.231537 15243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:36:11.231799 15243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:36:11.231861 15243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:36:11.235914 15243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:36:11.235972 15243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:36:11.236049 15243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:36:11.241317 15243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45151
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43069
--tserver_master_addrs=127.0.98.62:36259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:36:11.242355 15243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:36:11.243319 15243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:36:11.252436 15282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:36:11.252537 15243 server_base.cc:1034] running on GCE node
W20260927 02:36:11.252457 15285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:36:11.252718 15283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:36:11.253144 15243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:36:11.253810 15243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:36:11.254999 15243 hybrid_clock.cc:648] HybridClock initialized: now 1790476571254973 us; error 44 us; skew 500 ppm
Sep 27 02:36:11 dist-test-slave-grlk krb5kdc[14097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:36:11.257634 15243 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:36:11.258467 15243 webserver.cc:460] Webserver started at http://127.0.98.1:43069/ using document root <none> and password file <none>
I20260927 02:36:11.258991 15243 fs_manager.cc:359] Metadata directory not provided
I20260927 02:36:11.259055 15243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:36:11.262794 15243 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:36:11.264853 15292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:11.266201 15243 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260927 02:36:11.266311 15243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-0/wal
uuid: "d978229ba60143278b1226ab8132033d"
format_stamp: "Formatted at 2026-09-27 02:36:08 on dist-test-slave-grlk"
server_key: "667717698431fe3ce035658e6d4e7f21"
server_key_iv: "96c13820116b71566586b6eeec4bc1f8"
server_key_version: "encryptionkey@0"
I20260927 02:36:11.267010 15243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:36:11.300391 15243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:36:11.303402 15243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:36:11.303596 15243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:36:11.304173 15243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:36:11.305297 15299 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 02:36:11.310446 15243 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 02:36:11.310541 15243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260927 02:36:11.310613 15243 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 02:36:11.312317 15243 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 02:36:11.312367 15243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 02:36:11.312638 15299 tablet_bootstrap.cc:492] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d: Bootstrap starting.
I20260927 02:36:11.321753 15243 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45151
I20260927 02:36:11.321834 15406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45151 every 8 connection(s)
I20260927 02:36:11.322634 15243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:36:11.324348 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 15243
I20260927 02:36:11.324800 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34453
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36259
--webserver_port=40601
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:36:11.332279 15299 log.cc:824] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d: Log is configured to *not* fsync() on all Append() calls
I20260927 02:36:11.338797 15299 tablet_bootstrap.cc:492] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:36:11.339115 15299 tablet_bootstrap.cc:492] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d: Bootstrap complete.
I20260927 02:36:11.339970 15299 ts_tablet_manager.cc:1397] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d: Time spent bootstrapping tablet: real 0.028s user 0.011s sys 0.013s
I20260927 02:36:11.342206 15299 raft_consensus.cc:348] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b950ed19247482b9e0ff9b7a7d9eac1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39691 } } peers { permanent_uuid: "64a55ced77b1438e87fbb5e5feb28738" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34453 } } peers { permanent_uuid: "d978229ba60143278b1226ab8132033d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45151 } }
I20260927 02:36:11.342578 15299 raft_consensus.cc:729] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d978229ba60143278b1226ab8132033d, State: Initialized, Role: FOLLOWER
I20260927 02:36:11.343003 15299 consensus_queue.cc:260] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b950ed19247482b9e0ff9b7a7d9eac1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39691 } } peers { permanent_uuid: "64a55ced77b1438e87fbb5e5feb28738" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34453 } } peers { permanent_uuid: "d978229ba60143278b1226ab8132033d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45151 } }
I20260927 02:36:11.343827 15299 ts_tablet_manager.cc:1428] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d: Time spent starting tablet: real 0.004s user 0.006s sys 0.001s
W20260927 02:36:11.380790 15409 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:36:11.324539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.1:41373)
0927 02:36:11.325110 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:11.325186 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:11.326252 (+ 1066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:11.379880 (+ 53628us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":144,"threads_started":1}
W20260927 02:36:11.381390 15407 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36259 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36259: Client connection negotiation failed: client connection to 127.0.98.62:36259: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 02:36:11.382092 15409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:11.381829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.1:49311)
0927 02:36:11.381958 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:11.382029 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 02:36:11.382689 15409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:11.382496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.1:53451)
0927 02:36:11.382590 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:11.382636 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 02:36:11.382838 15407 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 02:36:11.452540 15410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:36:11.452773 15410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:36:11.452814 15410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:36:11.456880 15410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:36:11.456991 15410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:36:11.457070 15410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:36:11.462276 15410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34453
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=40601
--tserver_master_addrs=127.0.98.62:36259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:36:11.463223 15410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:36:11.464170 15410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:36:11.473177 15418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:36:11.473206 15420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:36:11.473187 15417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:36:11.473403 15410 server_base.cc:1034] running on GCE node
I20260927 02:36:11.473789 15410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:36:11.474406 15410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:36:11.475572 15410 hybrid_clock.cc:648] HybridClock initialized: now 1790476571475546 us; error 42 us; skew 500 ppm
Sep 27 02:36:11 dist-test-slave-grlk krb5kdc[14097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:36:11.478148 15410 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:36:11.478976 15410 webserver.cc:460] Webserver started at http://127.0.98.2:40601/ using document root <none> and password file <none>
I20260927 02:36:11.479499 15410 fs_manager.cc:359] Metadata directory not provided
I20260927 02:36:11.479568 15410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:36:11.483244 15410 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:36:11.485225 15427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:11.486554 15410 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 02:36:11.486677 15410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-1/wal
uuid: "64a55ced77b1438e87fbb5e5feb28738"
format_stamp: "Formatted at 2026-09-27 02:36:09 on dist-test-slave-grlk"
server_key: "7bb7680e522f3ba3c3b126c4092962a7"
server_key_iv: "c2d0b22f8f814c8396a18878bcbd04dd"
server_key_version: "encryptionkey@0"
I20260927 02:36:11.487284 15410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:36:11.501638 15410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:36:11.504218 15410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:36:11.504351 15410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:36:11.504838 15410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:36:11.505810 15434 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260927 02:36:11.509330 15434 ts_tablet_manager.cc:1852] T 9b75852427204cc5a1a24b8e68734fce P 64a55ced77b1438e87fbb5e5feb28738: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260927 02:36:11.509706 15434 ts_tablet_manager.cc:1905] T 9b75852427204cc5a1a24b8e68734fce P 64a55ced77b1438e87fbb5e5feb28738: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:36:11.513233 15434 ts_tablet_manager.cc:1918] T 9b75852427204cc5a1a24b8e68734fce P 64a55ced77b1438e87fbb5e5feb28738: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:36:11.513326 15434 log.cc:1196] T 9b75852427204cc5a1a24b8e68734fce P 64a55ced77b1438e87fbb5e5feb28738: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-1/wal/wals/9b75852427204cc5a1a24b8e68734fce
I20260927 02:36:11.513597 15434 ts_tablet_manager.cc:1939] T 9b75852427204cc5a1a24b8e68734fce P 64a55ced77b1438e87fbb5e5feb28738: Deleting consensus metadata
I20260927 02:36:11.513870 15410 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260927 02:36:11.513928 15410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.000s sys 0.000s
I20260927 02:36:11.513991 15410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:36:11.514020 15410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:11.522428 15410 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34453
I20260927 02:36:11.522500 15541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34453 every 8 connection(s)
I20260927 02:36:11.523308 15410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:36:11.526799 15544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:11.525350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.2:40161)
0927 02:36:11.525850 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:11.525963 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":115,"threads_started":1}
W20260927 02:36:11.531656 15542 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36259 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36259: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
I20260927 02:36:11.532130 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 15410
I20260927 02:36:11.532483 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39691
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36259
--webserver_port=42463
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:36:11.533030 15544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:11.532174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.2:32805)
0927 02:36:11.532860 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:11.532911 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":607}
I20260927 02:36:11.533607 15544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:11.533406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.2:59719)
0927 02:36:11.533510 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:11.533555 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 02:36:11.533762 15542 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 02:36:11.636765 15545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:36:11.637042 15545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:36:11.637089 15545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:36:11.640880 15545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:36:11.640923 15545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:36:11.641021 15545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:36:11.645995 15545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39691
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42463
--tserver_master_addrs=127.0.98.62:36259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:36:11.646929 15545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:36:11.647878 15545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:36:11.656658 15553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:36:11.656656 15550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:36:11.656679 15551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:36:11.656914 15545 server_base.cc:1034] running on GCE node
I20260927 02:36:11.657331 15545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:36:11.657948 15545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:36:11.659096 15545 hybrid_clock.cc:648] HybridClock initialized: now 1790476571659074 us; error 36 us; skew 500 ppm
Sep 27 02:36:11 dist-test-slave-grlk krb5kdc[14097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:36:11.661497 15545 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:36:11.662271 15545 webserver.cc:460] Webserver started at http://127.0.98.3:42463/ using document root <none> and password file <none>
I20260927 02:36:11.662763 15545 fs_manager.cc:359] Metadata directory not provided
I20260927 02:36:11.662827 15545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:36:11.666313 15545 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:36:11.668124 15560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:11.669229 15545 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:36:11.669333 15545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-2/wal
uuid: "0b950ed19247482b9e0ff9b7a7d9eac1"
format_stamp: "Formatted at 2026-09-27 02:36:09 on dist-test-slave-grlk"
server_key: "96a4282c3ae44acfb5d64bd5a83d3b57"
server_key_iv: "bcddd3a4e526610621bd2bb38e779a47"
server_key_version: "encryptionkey@0"
I20260927 02:36:11.669827 15545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:36:11.709797 15545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:36:11.712852 15545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:36:11.713057 15545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:36:11.713639 15545 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:36:11.714481 15545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:36:11.714526 15545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:11.714577 15545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:36:11.714605 15545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:11.722800 15545 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39691
I20260927 02:36:11.722831 15673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39691 every 8 connection(s)
I20260927 02:36:11.723793 15545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:36:11.727439 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:11.725774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.3:52977)
0927 02:36:11.726230 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:11.726331 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":118,"threads_started":1}
W20260927 02:36:11.727845 15674 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36259 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36259: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
I20260927 02:36:11.728372 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:11.728161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.3:58005)
0927 02:36:11.728263 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:11.728313 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 02:36:11.728376 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 15545
I20260927 02:36:11.728921 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:11.728748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.3:42377)
0927 02:36:11.728839 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:11.728877 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 02:36:11.729144 15674 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 02:36:12.383788 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:12.383377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.1:52953)
0927 02:36:12.383620 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:12.383679 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":72,"threads_started":1}
I20260927 02:36:12.534667 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:12.534287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.2:52093)
0927 02:36:12.534529 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:12.534586 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":64,"threads_started":1}
I20260927 02:36:12.712664 15680 raft_consensus.cc:482] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:36:12.712852 15680 raft_consensus.cc:504] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b950ed19247482b9e0ff9b7a7d9eac1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39691 } } peers { permanent_uuid: "64a55ced77b1438e87fbb5e5feb28738" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34453 } } peers { permanent_uuid: "d978229ba60143278b1226ab8132033d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45151 } }
I20260927 02:36:12.713972 15680 leader_election.cc:288] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b950ed19247482b9e0ff9b7a7d9eac1 (127.0.98.3:39691), 64a55ced77b1438e87fbb5e5feb28738 (127.0.98.2:34453)
Sep 27 02:36:12 dist-test-slave-grlk krb5kdc[14097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 27 02:36:12 dist-test-slave-grlk krb5kdc[14097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260927 02:36:12.729674 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:12.714506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34453 (local address 127.0.98.1:37155)
0927 02:36:12.714837 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:12.714854 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:12.714958 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:12.717028 (+ 2070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:36:12.717031 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:36:12.717525 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:36:12.718010 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:12.718031 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:12.719905 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:12.719909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:12.721179 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:12.721189 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:12.721917 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:12.721920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:12.722081 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:12.722774 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:36:12.722786 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:36:12.725002 (+ 2216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:36:12.727581 (+ 2579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:12.727584 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:12.727586 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:12.727831 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:12.728188 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:12.728191 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:12.728193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:12.728336 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:12.728959 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:36:12.728965 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:36:12.729154 (+ 189us) client_negotiation.cc:769] Sending connection context
0927 02:36:12.729311 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 02:36:12.729547 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":4,"spinlock_wait_cycles":2432,"thread_start_us":79,"threads_started":1}
I20260927 02:36:12.729998 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:12.729604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.3:53007)
0927 02:36:12.729864 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:12.729908 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":66,"threads_started":1}
I20260927 02:36:12.730237 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:12.714204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39691 (local address 127.0.98.1:53787)
0927 02:36:12.715076 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:12.715084 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:12.715175 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:12.717010 (+ 1835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:36:12.717016 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:36:12.718036 (+ 1020us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:36:12.718149 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:12.718156 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:12.720532 (+ 2376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:12.720536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:12.721298 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:12.721305 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:12.722031 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:12.722034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:12.722169 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:12.722732 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:36:12.722751 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:36:12.724584 (+ 1833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:36:12.727435 (+ 2851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:12.727439 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:12.727450 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:12.727803 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:12.728159 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:12.728162 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:12.728164 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:12.728326 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:12.729799 (+ 1473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:36:12.729802 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:36:12.729894 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 02:36:12.729988 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 02:36:12.730099 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"mutex_wait_us":36}
I20260927 02:36:12.730340 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:12.714443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37155 (local address 127.0.98.2:34453)
0927 02:36:12.715061 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:12.715077 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:12.715120 (+ 43us) server_negotiation.cc:407] Connection header received
0927 02:36:12.716355 (+ 1235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:12.716365 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:12.716748 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:12.717115 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:36:12.718199 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:12.719774 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:12.721312 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:12.721800 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:12.721922 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:12.725148 (+ 3226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:36:12.725153 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:36:12.725163 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:36:12.725188 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:36:12.727489 (+ 2301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:12.727922 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:12.727926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:12.727931 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:12.728066 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:12.728448 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:12.728451 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:12.728453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:12.728814 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:36:12.728924 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:12.729387 (+ 463us) server_negotiation.cc:299] Negotiation successful
0927 02:36:12.729638 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":94,"threads_started":1}
I20260927 02:36:12.730830 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:12.714421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53787 (local address 127.0.98.3:39691)
0927 02:36:12.714957 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:12.714972 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:12.715264 (+ 292us) server_negotiation.cc:407] Connection header received
0927 02:36:12.716107 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:12.716116 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:12.716574 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:12.716877 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:36:12.718262 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:12.720377 (+ 2115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:12.721417 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:12.721907 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:12.722057 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:12.725133 (+ 3076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:36:12.725141 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:36:12.725156 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:36:12.725186 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:36:12.727325 (+ 2139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:12.727919 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:12.727925 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:12.727930 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:12.728066 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:12.728428 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:12.728431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:12.728433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:12.728815 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:36:12.728948 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:12.730490 (+ 1542us) server_negotiation.cc:299] Negotiation successful
0927 02:36:12.730713 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":84,"threads_started":1}
W20260927 02:36:12.732653 15293 leader_election.cc:341] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 64a55ced77b1438e87fbb5e5feb28738 (127.0.98.2:34453): Not found: Tablet not found: 9b75852427204cc5a1a24b8e68734fce
W20260927 02:36:12.732790 15294 leader_election.cc:341] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0b950ed19247482b9e0ff9b7a7d9eac1 (127.0.98.3:39691): Not found: Tablet not found: 9b75852427204cc5a1a24b8e68734fce
I20260927 02:36:12.732874 15294 leader_election.cc:302] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d978229ba60143278b1226ab8132033d; no voters: 0b950ed19247482b9e0ff9b7a7d9eac1, 64a55ced77b1438e87fbb5e5feb28738
I20260927 02:36:12.733130 15680 raft_consensus.cc:2726] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 02:36:13.385212 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:13.384691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.1:39739)
0927 02:36:13.385010 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:13.385088 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":81,"threads_started":1}
I20260927 02:36:13.535696 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:13.535326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.2:49107)
0927 02:36:13.535548 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:13.535601 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":73,"threads_started":1}
I20260927 02:36:13.731139 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:13.730680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.3:50111)
0927 02:36:13.730951 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:13.731015 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":69,"threads_started":1}
I20260927 02:36:14.386344 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:14.385936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.1:51719)
0927 02:36:14.386184 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:14.386245 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":70,"threads_started":1}
I20260927 02:36:14.536684 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:14.536321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.2:43147)
0927 02:36:14.536549 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:14.536599 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":80,"threads_started":1}
I20260927 02:36:14.732244 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:14.731831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.3:38175)
0927 02:36:14.732095 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:14.732152 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":82,"threads_started":1}
I20260927 02:36:14.979136 15691 raft_consensus.cc:482] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:36:14.979271 15691 raft_consensus.cc:504] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b950ed19247482b9e0ff9b7a7d9eac1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39691 } } peers { permanent_uuid: "64a55ced77b1438e87fbb5e5feb28738" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34453 } } peers { permanent_uuid: "d978229ba60143278b1226ab8132033d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45151 } }
I20260927 02:36:14.979673 15691 leader_election.cc:288] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b950ed19247482b9e0ff9b7a7d9eac1 (127.0.98.3:39691), 64a55ced77b1438e87fbb5e5feb28738 (127.0.98.2:34453)
W20260927 02:36:14.980398 15294 leader_election.cc:341] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0b950ed19247482b9e0ff9b7a7d9eac1 (127.0.98.3:39691): Not found: Tablet not found: 9b75852427204cc5a1a24b8e68734fce
W20260927 02:36:14.980505 15293 leader_election.cc:341] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 64a55ced77b1438e87fbb5e5feb28738 (127.0.98.2:34453): Not found: Tablet not found: 9b75852427204cc5a1a24b8e68734fce
I20260927 02:36:14.980590 15293 leader_election.cc:302] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d978229ba60143278b1226ab8132033d; no voters: 0b950ed19247482b9e0ff9b7a7d9eac1, 64a55ced77b1438e87fbb5e5feb28738
I20260927 02:36:14.980737 15691 raft_consensus.cc:2726] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 02:36:15.387468 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:15.387046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.1:40865)
0927 02:36:15.387292 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:15.387351 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":70,"threads_started":1}
I20260927 02:36:15.537781 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:15.537367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.2:49309)
0927 02:36:15.537622 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:15.537673 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":83,"threads_started":1}
I20260927 02:36:15.733335 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:15.732966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.3:53885)
0927 02:36:15.733197 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:15.733249 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":83,"threads_started":1}
I20260927 02:36:16.388820 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:16.388349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.1:54249)
0927 02:36:16.388664 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:16.388724 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":94,"threads_started":1}
I20260927 02:36:16.538930 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:16.538534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.2:39489)
0927 02:36:16.538775 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:16.538833 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":89,"threads_started":1}
I20260927 02:36:16.734495 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:16.734033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.3:57087)
0927 02:36:16.734327 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:16.734386 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":118,"threads_started":1}
I20260927 02:36:17.389966 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:17.389545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.1:34545)
0927 02:36:17.389789 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:17.389866 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":88,"threads_started":1}
I20260927 02:36:17.540023 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:17.539656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.2:35993)
0927 02:36:17.539870 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:17.539923 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":66,"threads_started":1}
I20260927 02:36:17.735494 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:17.735125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.3:47483)
0927 02:36:17.735343 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:17.735393 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":61,"threads_started":1}
I20260927 02:36:18.243126 15701 raft_consensus.cc:482] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:36:18.243259 15701 raft_consensus.cc:504] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b950ed19247482b9e0ff9b7a7d9eac1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39691 } } peers { permanent_uuid: "64a55ced77b1438e87fbb5e5feb28738" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34453 } } peers { permanent_uuid: "d978229ba60143278b1226ab8132033d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45151 } }
I20260927 02:36:18.243635 15701 leader_election.cc:288] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b950ed19247482b9e0ff9b7a7d9eac1 (127.0.98.3:39691), 64a55ced77b1438e87fbb5e5feb28738 (127.0.98.2:34453)
W20260927 02:36:18.244352 15294 leader_election.cc:341] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0b950ed19247482b9e0ff9b7a7d9eac1 (127.0.98.3:39691): Not found: Tablet not found: 9b75852427204cc5a1a24b8e68734fce
W20260927 02:36:18.244454 15293 leader_election.cc:341] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 64a55ced77b1438e87fbb5e5feb28738 (127.0.98.2:34453): Not found: Tablet not found: 9b75852427204cc5a1a24b8e68734fce
I20260927 02:36:18.244535 15293 leader_election.cc:302] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d978229ba60143278b1226ab8132033d; no voters: 0b950ed19247482b9e0ff9b7a7d9eac1, 64a55ced77b1438e87fbb5e5feb28738
I20260927 02:36:18.244686 15701 raft_consensus.cc:2726] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 02:36:18.390986 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:18.390611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.1:37713)
0927 02:36:18.390839 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:18.390889 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":78,"threads_started":1}
I20260927 02:36:18.541222 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:18.540760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.2:46111)
0927 02:36:18.541079 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:18.541130 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":73,"threads_started":1}
I20260927 02:36:18.736511 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:18.736111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.3:55391)
0927 02:36:18.736372 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:18.736422 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":62,"threads_started":1}
I20260927 02:36:19.392064 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:19.391647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.1:45311)
0927 02:36:19.391902 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:19.391959 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":72,"threads_started":1}
I20260927 02:36:19.542364 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:19.541961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.2:59881)
0927 02:36:19.542209 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:19.542265 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":81,"threads_started":1}
I20260927 02:36:19.737831 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:19.737357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.3:43831)
0927 02:36:19.737634 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:19.737718 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":89,"threads_started":1}
I20260927 02:36:20.393155 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:20.392725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.1:41143)
0927 02:36:20.392984 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:20.393043 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":70,"threads_started":1}
I20260927 02:36:20.543329 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:20.542973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.2:42999)
0927 02:36:20.543191 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:20.543242 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":70,"threads_started":1}
I20260927 02:36:20.738916 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:20.738514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.3:35915)
0927 02:36:20.738767 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:20.738823 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":74,"threads_started":1}
I20260927 02:36:21.394377 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:21.393960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.1:45275)
0927 02:36:21.394227 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:21.394282 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":72,"threads_started":1}
I20260927 02:36:21.544478 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:21.544090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.2:59985)
0927 02:36:21.544324 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:21.544380 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":67,"threads_started":1}
I20260927 02:36:21.739985 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:21.739602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.3:50915)
0927 02:36:21.739840 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:21.739893 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":79,"threads_started":1}
I20260927 02:36:22.395540 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:22.395143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.1:58311)
0927 02:36:22.395389 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:22.395445 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":83,"threads_started":1}
I20260927 02:36:22.545543 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:22.545168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.2:37227)
0927 02:36:22.545402 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:22.545451 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":81,"threads_started":1}
I20260927 02:36:22.741112 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:22.740694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.3:45773)
0927 02:36:22.740926 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:22.741011 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":61,"threads_started":1}
I20260927 02:36:22.790807 15717 raft_consensus.cc:482] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:36:22.790934 15717 raft_consensus.cc:504] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b950ed19247482b9e0ff9b7a7d9eac1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39691 } } peers { permanent_uuid: "64a55ced77b1438e87fbb5e5feb28738" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34453 } } peers { permanent_uuid: "d978229ba60143278b1226ab8132033d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45151 } }
I20260927 02:36:22.791324 15717 leader_election.cc:288] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b950ed19247482b9e0ff9b7a7d9eac1 (127.0.98.3:39691), 64a55ced77b1438e87fbb5e5feb28738 (127.0.98.2:34453)
W20260927 02:36:22.791960 15294 leader_election.cc:341] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0b950ed19247482b9e0ff9b7a7d9eac1 (127.0.98.3:39691): Not found: Tablet not found: 9b75852427204cc5a1a24b8e68734fce
W20260927 02:36:22.792057 15293 leader_election.cc:341] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 64a55ced77b1438e87fbb5e5feb28738 (127.0.98.2:34453): Not found: Tablet not found: 9b75852427204cc5a1a24b8e68734fce
I20260927 02:36:22.792151 15293 leader_election.cc:302] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d978229ba60143278b1226ab8132033d; no voters: 0b950ed19247482b9e0ff9b7a7d9eac1, 64a55ced77b1438e87fbb5e5feb28738
I20260927 02:36:22.792289 15717 raft_consensus.cc:2726] T 9b75852427204cc5a1a24b8e68734fce P d978229ba60143278b1226ab8132033d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 02:36:23.396623 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:23.396239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.1:55291)
0927 02:36:23.396472 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:23.396525 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":73,"threads_started":1}
I20260927 02:36:23.546589 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:23.546210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.2:59005)
0927 02:36:23.546438 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:23.546490 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":75,"threads_started":1}
I20260927 02:36:23.742278 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:23.741863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.3:50077)
0927 02:36:23.742131 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:23.742188 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":71,"threads_started":1}
I20260927 02:36:24.397873 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:24.397400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.1:40931)
0927 02:36:24.397702 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:24.397772 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":85,"threads_started":1}
I20260927 02:36:24.547834 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:24.547358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.2:58535)
0927 02:36:24.547655 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:24.547726 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":88,"threads_started":1}
I20260927 02:36:24.743432 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:24.743037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.3:49697)
0927 02:36:24.743284 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:24.743340 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":71,"threads_started":1}
I20260927 02:36:25.399075 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:25.398592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.1:51025)
0927 02:36:25.398884 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:25.398970 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":77,"threads_started":1}
I20260927 02:36:25.549208 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:25.548752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.2:54387)
0927 02:36:25.549054 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:25.549114 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":90,"threads_started":1}
I20260927 02:36:25.744518 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:25.744107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.3:45543)
0927 02:36:25.744368 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:25.744426 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":75,"threads_started":1}
I20260927 02:36:26.400293 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:26.399930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.1:35943)
0927 02:36:26.400154 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:26.400203 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":69,"threads_started":1}
I20260927 02:36:26.550453 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:26.549977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36259 (local address 127.0.98.2:49867)
0927 02:36:26.550255 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:26.550321 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":71,"threads_started":1}
I20260927 02:36:26.731750 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 15243
I20260927 02:36:26.736753 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 15410
I20260927 02:36:26.741370 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 15545
2026-09-27T02:36:26Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19334 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:36:26 dist-test-slave-grlk krb5kdc[15731](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:36:26 dist-test-slave-grlk krb5kdc[15731](info): set up 2 sockets
Sep 27 02:36:26 dist-test-slave-grlk krb5kdc[15731](info): commencing operation
krb5kdc: starting...
W20260927 02:36:27.704968 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/test-user.keytab.
Sep 27 02:36:27 dist-test-slave-grlk krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:36:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:36:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:36:27.820699 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39525
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39525
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:36:27.935678 15744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:36:27.935931 15744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:36:27.935971 15744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:36:27.940155 15744 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 02:36:27.940230 15744 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 02:36:27.940245 15744 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 02:36:27.940258 15744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:36:27.940275 15744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:36:27.940292 15744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:36:27.940305 15744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:36:27.940328 15744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:36:27.945727 15744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:36:27.946759 15744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:36:27.947722 15744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:36:27.956278 15749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:36:27.956355 15744 server_base.cc:1034] running on GCE node
W20260927 02:36:27.956352 15750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:36:27.956269 15752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:36:27.957046 15744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:36:27.958092 15744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:36:27.959242 15744 hybrid_clock.cc:648] HybridClock initialized: now 1790476587959231 us; error 29 us; skew 500 ppm
Sep 27 02:36:27 dist-test-slave-grlk krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:36:27.961953 15744 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:36:27.962865 15744 webserver.cc:460] Webserver started at http://127.0.98.62:35165/ using document root <none> and password file <none>
I20260927 02:36:27.963398 15744 fs_manager.cc:359] Metadata directory not provided
I20260927 02:36:27.963461 15744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:36:27.963651 15744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:36:27.965679 15744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "4f19bae86c5b42b0b6a725e5b1b20836"
format_stamp: "Formatted at 2026-09-27 02:36:27 on dist-test-slave-grlk"
server_key: "b0699c1d4624bea5a4f5c11aa99316e2"
server_key_iv: "2fa3876119f1a2b575804cc2a56b3967"
server_key_version: "encryptionkey@0"
I20260927 02:36:27.966198 15744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "4f19bae86c5b42b0b6a725e5b1b20836"
format_stamp: "Formatted at 2026-09-27 02:36:27 on dist-test-slave-grlk"
server_key: "b0699c1d4624bea5a4f5c11aa99316e2"
server_key_iv: "2fa3876119f1a2b575804cc2a56b3967"
server_key_version: "encryptionkey@0"
I20260927 02:36:27.969805 15744 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 02:36:27.972159 15759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:27.973338 15744 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260927 02:36:27.973464 15744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "4f19bae86c5b42b0b6a725e5b1b20836"
format_stamp: "Formatted at 2026-09-27 02:36:27 on dist-test-slave-grlk"
server_key: "b0699c1d4624bea5a4f5c11aa99316e2"
server_key_iv: "2fa3876119f1a2b575804cc2a56b3967"
server_key_version: "encryptionkey@0"
I20260927 02:36:27.973564 15744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:36:27.988914 15744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:36:27.992066 15744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:36:27.992255 15744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:36:27.998801 15744 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39525
I20260927 02:36:27.999044 15811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39525 every 8 connection(s)
I20260927 02:36:27.999708 15744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:36:28.002274 15812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:36:28.007593 15812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836: Bootstrap starting.
I20260927 02:36:28.007902 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 15744
I20260927 02:36:28.008013 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:36:28.008263 392 external_mini_cluster.cc:1428] Setting key 9a43b6376c0e948f8edfeb3083b93cc8
I20260927 02:36:28.010106 15812 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836: Neither blocks nor log segments found. Creating new log.
I20260927 02:36:28.010794 15812 log.cc:824] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836: Log is configured to *not* fsync() on all Append() calls
I20260927 02:36:28.012758 15812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836: No bootstrap required, opened a new log
Sep 27 02:36:28 dist-test-slave-grlk krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:36:28.016799 15812 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f19bae86c5b42b0b6a725e5b1b20836" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39525 } }
I20260927 02:36:28.017040 15812 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:36:28.017096 15812 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f19bae86c5b42b0b6a725e5b1b20836, State: Initialized, Role: FOLLOWER
I20260927 02:36:28.017552 15812 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f19bae86c5b42b0b6a725e5b1b20836" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39525 } }
I20260927 02:36:28.017691 15812 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:36:28.017752 15812 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:36:28.017850 15812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:36:28.018765 15812 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f19bae86c5b42b0b6a725e5b1b20836" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39525 } }
I20260927 02:36:28.019055 15812 leader_election.cc:302] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f19bae86c5b42b0b6a725e5b1b20836; no voters:
I20260927 02:36:28.019338 15812 leader_election.cc:288] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:36:28.019388 15817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:36:28.019611 15817 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [term 1 LEADER]: Becoming Leader. State: Replica: 4f19bae86c5b42b0b6a725e5b1b20836, State: Running, Role: LEADER
I20260927 02:36:28.019953 15817 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f19bae86c5b42b0b6a725e5b1b20836" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39525 } }
I20260927 02:36:28.020517 15812 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:36:28.021520 15818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f19bae86c5b42b0b6a725e5b1b20836" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f19bae86c5b42b0b6a725e5b1b20836" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39525 } } }
I20260927 02:36:28.021631 15818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [sys.catalog]: This master's current role is: LEADER
I20260927 02:36:28.021950 15819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f19bae86c5b42b0b6a725e5b1b20836. Latest consensus state: current_term: 1 leader_uuid: "4f19bae86c5b42b0b6a725e5b1b20836" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f19bae86c5b42b0b6a725e5b1b20836" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39525 } } }
I20260927 02:36:28.022001 15826 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:36:28.022068 15819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [sys.catalog]: This master's current role is: LEADER
I20260927 02:36:28.022410 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:28.009529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52146 (local address 127.0.98.62:39525)
0927 02:36:28.009992 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:28.010007 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:28.010040 (+ 33us) server_negotiation.cc:407] Connection header received
0927 02:36:28.010696 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:28.010714 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:28.011058 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:28.011408 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:36:28.012168 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:28.013196 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:28.014309 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:28.014760 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:28.014863 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:28.017479 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:36:28.017483 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:36:28.017496 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:36:28.017520 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:36:28.019155 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:28.019627 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:28.019633 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:28.019639 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:28.019778 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:28.020056 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:28.020060 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:28.020062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:28.020528 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:36:28.020716 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:28.021142 (+ 426us) server_negotiation.cc:299] Negotiation successful
0927 02:36:28.021449 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":127,"threads_started":1}
I20260927 02:36:28.025089 15826 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:36:28.029846 15826 catalog_manager.cc:1347] Generated new cluster ID: 6620759ac746484aa0f68da98198dbed
I20260927 02:36:28.029906 15826 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:36:28.065140 15826 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:36:28.066126 15826 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:36:28.070467 15826 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836: Generated new TSK 0
I20260927 02:36:28.071097 15826 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:36:28.099393 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39525
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:36:28.227551 15838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:36:28.227805 15838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:36:28.227850 15838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:36:28.231814 15838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:36:28.231874 15838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:36:28.231942 15838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:36:28.237088 15838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:36:28.238010 15838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:36:28.239064 15838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:36:28.248200 15844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:36:28.248360 15838 server_base.cc:1034] running on GCE node
W20260927 02:36:28.248257 15843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:36:28.248261 15846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:36:28.248809 15838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:36:28.249413 15838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:36:28.250554 15838 hybrid_clock.cc:648] HybridClock initialized: now 1790476588250533 us; error 33 us; skew 500 ppm
Sep 27 02:36:28 dist-test-slave-grlk krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:36:28.253118 15838 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:36:28.254053 15838 webserver.cc:460] Webserver started at http://127.0.98.1:45407/ using document root <none> and password file <none>
I20260927 02:36:28.254616 15838 fs_manager.cc:359] Metadata directory not provided
I20260927 02:36:28.254675 15838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:36:28.254861 15838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:36:28.256745 15838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-0/data/instance:
uuid: "bcf36ad493654e22a988b3d5fa661f10"
format_stamp: "Formatted at 2026-09-27 02:36:28 on dist-test-slave-grlk"
server_key: "bd1dae9cfba5669d741e56ce9325ef77"
server_key_iv: "845aaf2fe601841a23a95204480fecfd"
server_key_version: "encryptionkey@0"
I20260927 02:36:28.257311 15838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bcf36ad493654e22a988b3d5fa661f10"
format_stamp: "Formatted at 2026-09-27 02:36:28 on dist-test-slave-grlk"
server_key: "bd1dae9cfba5669d741e56ce9325ef77"
server_key_iv: "845aaf2fe601841a23a95204480fecfd"
server_key_version: "encryptionkey@0"
I20260927 02:36:28.260854 15838 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:36:28.263342 15853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:28.264536 15838 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 02:36:28.264655 15838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-0/wal
uuid: "bcf36ad493654e22a988b3d5fa661f10"
format_stamp: "Formatted at 2026-09-27 02:36:28 on dist-test-slave-grlk"
server_key: "bd1dae9cfba5669d741e56ce9325ef77"
server_key_iv: "845aaf2fe601841a23a95204480fecfd"
server_key_version: "encryptionkey@0"
I20260927 02:36:28.264760 15838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:36:28.280367 15838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:36:28.283505 15838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:36:28.283691 15838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:36:28.284242 15838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:36:28.285096 15838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:36:28.285143 15838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:28.285195 15838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:36:28.285224 15838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:28.293712 15838 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46463
I20260927 02:36:28.293740 15966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46463 every 8 connection(s)
I20260927 02:36:28.294531 15838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:36:28.295719 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 15838
I20260927 02:36:28.295848 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:36:28.296150 392 external_mini_cluster.cc:1428] Setting key 973784b6d18f4cb75e347ce4b90fc55d
Sep 27 02:36:28 dist-test-slave-grlk krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 02:36:28.307525 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:28.296185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46093 (local address 127.0.98.62:39525)
0927 02:36:28.296309 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:28.296312 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:28.297257 (+ 945us) server_negotiation.cc:407] Connection header received
0927 02:36:28.298548 (+ 1291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:28.298552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:28.298602 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:28.298677 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:36:28.299755 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:28.300256 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:28.301457 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:28.301684 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:28.301768 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:28.304164 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:36:28.304167 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:36:28.304169 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:36:28.304192 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:36:28.305486 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:28.305973 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:28.305978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:28.305980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:28.306064 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:28.306398 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:28.306402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:28.306403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:28.306642 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:36:28.306755 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:28.307204 (+ 449us) server_negotiation.cc:299] Negotiation successful
0927 02:36:28.307340 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 02:36:28.308295 15969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:28.296625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.1:46093)
0927 02:36:28.297103 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:28.297162 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:28.298355 (+ 1193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:28.298849 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:36:28.298856 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:36:28.299183 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:36:28.299621 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:28.299634 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:28.300370 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:28.300374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:28.301347 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:28.301357 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:28.301810 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:28.301814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:28.301969 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:28.302571 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:36:28.302588 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:36:28.304008 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:36:28.305601 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:28.305608 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:28.305621 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:28.305864 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:28.306149 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:28.306153 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:28.306155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:28.306304 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:28.306779 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:36:28.306784 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:36:28.306986 (+ 202us) client_negotiation.cc:769] Sending connection context
0927 02:36:28.307145 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 02:36:28.307375 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":108,"threads_started":1}
I20260927 02:36:28.309738 15967 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39525
I20260927 02:36:28.309973 15967 heartbeater.cc:464] Registering TS with master...
I20260927 02:36:28.310514 15967 heartbeater.cc:511] Master 127.0.98.62:39525 requested a full tablet report, sending...
I20260927 02:36:28.312163 15776 ts_manager.cc:194] Registered new tserver with Master: bcf36ad493654e22a988b3d5fa661f10 (127.0.98.1:46463)
I20260927 02:36:28.313694 15776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46093
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:36:28.321326 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39525
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:36:28.437872 15972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:36:28.438107 15972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:36:28.438149 15972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:36:28.442031 15972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:36:28.442090 15972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:36:28.442147 15972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:36:28.447649 15972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:36:28.448565 15972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:36:28.449508 15972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:36:28.458662 15980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:36:28.458681 15977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:36:28.458681 15978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:36:28.458729 15972 server_base.cc:1034] running on GCE node
I20260927 02:36:28.459223 15972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:36:28.459785 15972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:36:28.460981 15972 hybrid_clock.cc:648] HybridClock initialized: now 1790476588460959 us; error 30 us; skew 500 ppm
Sep 27 02:36:28 dist-test-slave-grlk krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:36:28.463595 15972 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:36:28.464459 15972 webserver.cc:460] Webserver started at http://127.0.98.2:33353/ using document root <none> and password file <none>
I20260927 02:36:28.465016 15972 fs_manager.cc:359] Metadata directory not provided
I20260927 02:36:28.465083 15972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:36:28.465276 15972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:36:28.467228 15972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-1/data/instance:
uuid: "fb9f7e48bd2a42b0ad0cee0ee135f636"
format_stamp: "Formatted at 2026-09-27 02:36:28 on dist-test-slave-grlk"
server_key: "b572203f42735d94f070734bd05f9f55"
server_key_iv: "4767fea16b5ad0c79da83fc0524a3d1f"
server_key_version: "encryptionkey@0"
I20260927 02:36:28.467739 15972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fb9f7e48bd2a42b0ad0cee0ee135f636"
format_stamp: "Formatted at 2026-09-27 02:36:28 on dist-test-slave-grlk"
server_key: "b572203f42735d94f070734bd05f9f55"
server_key_iv: "4767fea16b5ad0c79da83fc0524a3d1f"
server_key_version: "encryptionkey@0"
I20260927 02:36:28.471197 15972 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:36:28.473443 15987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:28.474561 15972 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:36:28.474660 15972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-1/wal
uuid: "fb9f7e48bd2a42b0ad0cee0ee135f636"
format_stamp: "Formatted at 2026-09-27 02:36:28 on dist-test-slave-grlk"
server_key: "b572203f42735d94f070734bd05f9f55"
server_key_iv: "4767fea16b5ad0c79da83fc0524a3d1f"
server_key_version: "encryptionkey@0"
I20260927 02:36:28.474751 15972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:36:28.487993 15972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:36:28.490918 15972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:36:28.491094 15972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:36:28.491623 15972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:36:28.492384 15972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:36:28.492430 15972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:28.492479 15972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:36:28.492508 15972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:28.500728 15972 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40749
I20260927 02:36:28.500761 16100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40749 every 8 connection(s)
I20260927 02:36:28.501572 15972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:36:28.506431 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 15972
I20260927 02:36:28.506543 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:36:28.506789 392 external_mini_cluster.cc:1428] Setting key 9f580a15685977beda5a5961fa75b57f
Sep 27 02:36:28 dist-test-slave-grlk krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:36:28.513746 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:28.503176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35387 (local address 127.0.98.62:39525)
0927 02:36:28.503297 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:28.503301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:28.504028 (+ 727us) server_negotiation.cc:407] Connection header received
0927 02:36:28.505258 (+ 1230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:28.505262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:28.505315 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:28.505387 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:36:28.506442 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:28.507046 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:28.508282 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:28.508497 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:28.508586 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:28.510718 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:36:28.510722 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:36:28.510724 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:36:28.510748 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:36:28.511996 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:28.512423 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:28.512426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:28.512427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:28.512495 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:28.512783 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:28.512785 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:28.512787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:28.512967 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:36:28.513053 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:28.513473 (+ 420us) server_negotiation.cc:299] Negotiation successful
0927 02:36:28.513593 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 02:36:28.514443 16103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:28.503494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.2:35387)
0927 02:36:28.503898 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:28.503936 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:28.505068 (+ 1132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:28.505551 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:36:28.505557 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:36:28.505871 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:36:28.506308 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:28.506320 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:28.507156 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:28.507159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:28.508174 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:28.508183 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:28.508626 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:28.508630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:28.508774 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:28.509299 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:36:28.509314 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:36:28.510579 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:36:28.512095 (+ 1516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:28.512101 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:28.512113 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:28.512336 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:28.512579 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:28.512582 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:28.512584 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:28.512703 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:28.513058 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:36:28.513064 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:36:28.513272 (+ 208us) client_negotiation.cc:769] Sending connection context
0927 02:36:28.513421 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 02:36:28.513700 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":117,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:36:28.515467 16101 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39525
I20260927 02:36:28.515661 16101 heartbeater.cc:464] Registering TS with master...
I20260927 02:36:28.516058 16101 heartbeater.cc:511] Master 127.0.98.62:39525 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 02:36:28.516918 15776 ts_manager.cc:194] Registered new tserver with Master: fb9f7e48bd2a42b0ad0cee0ee135f636 (127.0.98.2:40749)
I20260927 02:36:28.517616 15776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35387
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:36:28.531124 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39525
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:36:28.635915 16106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:36:28.636143 16106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:36:28.636180 16106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:36:28.640036 16106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:36:28.640084 16106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:36:28.640136 16106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:36:28.645026 16106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:36:28.646003 16106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:36:28.646950 16106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:36:28.655926 16112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:36:28.655892 16114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:36:28.656060 16106 server_base.cc:1034] running on GCE node
W20260927 02:36:28.656086 16111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:36:28.656519 16106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:36:28.657109 16106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:36:28.658267 16106 hybrid_clock.cc:648] HybridClock initialized: now 1790476588658251 us; error 28 us; skew 500 ppm
Sep 27 02:36:28 dist-test-slave-grlk krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:36:28.660707 16106 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:36:28.661522 16106 webserver.cc:460] Webserver started at http://127.0.98.3:36991/ using document root <none> and password file <none>
I20260927 02:36:28.662022 16106 fs_manager.cc:359] Metadata directory not provided
I20260927 02:36:28.662083 16106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:36:28.662254 16106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:36:28.664096 16106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-2/data/instance:
uuid: "060b505faf464a789ccc22f1bf18fa17"
format_stamp: "Formatted at 2026-09-27 02:36:28 on dist-test-slave-grlk"
server_key: "b1f068eccd152be1f1018a26a8a75225"
server_key_iv: "299479cda5846f2ea95e8cbdacd13760"
server_key_version: "encryptionkey@0"
I20260927 02:36:28.664584 16106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-2/wal/instance:
uuid: "060b505faf464a789ccc22f1bf18fa17"
format_stamp: "Formatted at 2026-09-27 02:36:28 on dist-test-slave-grlk"
server_key: "b1f068eccd152be1f1018a26a8a75225"
server_key_iv: "299479cda5846f2ea95e8cbdacd13760"
server_key_version: "encryptionkey@0"
I20260927 02:36:28.667900 16106 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 02:36:28.670176 16121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:28.671258 16106 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:36:28.671355 16106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-2/wal
uuid: "060b505faf464a789ccc22f1bf18fa17"
format_stamp: "Formatted at 2026-09-27 02:36:28 on dist-test-slave-grlk"
server_key: "b1f068eccd152be1f1018a26a8a75225"
server_key_iv: "299479cda5846f2ea95e8cbdacd13760"
server_key_version: "encryptionkey@0"
I20260927 02:36:28.671443 16106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:36:28.681680 16106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:36:28.683979 16106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:36:28.684128 16106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:36:28.684607 16106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:36:28.685381 16106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:36:28.685426 16106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:28.685474 16106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:36:28.685503 16106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:28.693286 16106 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35341
I20260927 02:36:28.693315 16234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35341 every 8 connection(s)
I20260927 02:36:28.694070 16106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:36:28.696357 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 16106
I20260927 02:36:28.696491 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:36:28.696790 392 external_mini_cluster.cc:1428] Setting key 9bda42c6e73f01cbdb2ba00c828d780f
Sep 27 02:36:28 dist-test-slave-grlk krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:36:28.706104 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:28.695637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55059 (local address 127.0.98.62:39525)
0927 02:36:28.695760 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:28.695764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:28.696590 (+ 826us) server_negotiation.cc:407] Connection header received
0927 02:36:28.697917 (+ 1327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:28.697921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:28.697974 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:28.698042 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:36:28.699105 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:28.699626 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:28.700699 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:28.700912 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:28.701018 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:28.703095 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:36:28.703098 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:36:28.703100 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:36:28.703122 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:36:28.704334 (+ 1212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:28.704768 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:28.704772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:28.704773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:28.704841 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:28.705162 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:28.705165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:28.705167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:28.705324 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:36:28.705406 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:28.705837 (+ 431us) server_negotiation.cc:299] Negotiation successful
0927 02:36:28.705949 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 02:36:28.706770 16237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:28.695952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.3:55059)
0927 02:36:28.696393 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:28.696447 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:28.697704 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:28.698207 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:36:28.698213 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:36:28.698524 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:36:28.698980 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:28.698992 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:28.699734 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:28.699737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:28.700591 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:28.700600 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:28.701060 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:28.701063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:28.701214 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:28.701669 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:36:28.701684 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:36:28.702954 (+ 1270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:36:28.704436 (+ 1482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:28.704441 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:28.704453 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:28.704681 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:28.704918 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:28.704921 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:28.704923 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:28.705078 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:28.705415 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:36:28.705420 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:36:28.705632 (+ 212us) client_negotiation.cc:769] Sending connection context
0927 02:36:28.705780 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 02:36:28.706069 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":107,"threads_started":1}
I20260927 02:36:28.707715 16235 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39525
I20260927 02:36:28.707890 16235 heartbeater.cc:464] Registering TS with master...
I20260927 02:36:28.708276 16235 heartbeater.cc:511] Master 127.0.98.62:39525 requested a full tablet report, sending...
I20260927 02:36:28.709162 15776 ts_manager.cc:194] Registered new tserver with Master: 060b505faf464a789ccc22f1bf18fa17 (127.0.98.3:35341)
I20260927 02:36:28.709787 15776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55059
I20260927 02:36:28.710086 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:36:28.718595 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:28.712245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52160 (local address 127.0.98.62:39525)
0927 02:36:28.712380 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:28.712384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:28.712469 (+ 85us) server_negotiation.cc:407] Connection header received
0927 02:36:28.712568 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:28.712570 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:28.712606 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:28.712669 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:36:28.713379 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:28.713878 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:28.714651 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:28.714847 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:28.714913 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:28.716086 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:36:28.716092 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:36:28.716095 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:36:28.716115 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:36:28.717349 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:28.717706 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:28.717708 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:28.717710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:28.717774 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:28.717944 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:28.717946 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:28.717947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:28.718126 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:36:28.718220 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:28.718368 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 02:36:28.718450 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 02:36:28.721081 15776 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52160:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:36:28.724078 15776 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:36:28.737617 16250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:28.732252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54056 (local address 127.0.98.2:40749)
0927 02:36:28.732736 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:28.732748 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:28.732769 (+ 21us) server_negotiation.cc:407] Connection header received
0927 02:36:28.733400 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:28.733407 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:28.733577 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:28.733727 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:36:28.733876 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:28.734934 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:28.736211 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:28.737043 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:28.737144 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:28.737281 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:28.737431 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 02:36:28.737502 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":129,"threads_started":1}
I20260927 02:36:28.737661 16249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:28.731729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40749 (local address 127.0.0.1:54056)
0927 02:36:28.732553 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:28.732591 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:28.732732 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:28.733712 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:36:28.733715 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:36:28.733730 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:36:28.733806 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:28.733812 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:28.735049 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:28.735052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:28.736102 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:28.736111 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:28.737152 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:28.737155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:28.737266 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:28.737305 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 02:36:28.737409 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 02:36:28.737510 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"mutex_wait_us":43,"spinlock_wait_cycles":5248,"thread_start_us":78,"threads_started":1}
I20260927 02:36:28.738273 16246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:28.732283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46463 (local address 127.0.0.1:35592)
0927 02:36:28.732947 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:28.732959 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:28.733025 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:28.733587 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:36:28.733589 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:36:28.733602 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:36:28.733676 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:28.733681 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:28.734682 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:28.734686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:28.736211 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:28.736219 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:28.737888 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:28.737892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:28.737988 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:28.738011 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 02:36:28.738088 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 02:36:28.738151 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"mutex_wait_us":30,"thread_start_us":80,"threads_started":1}
I20260927 02:36:28.738353 16251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:28.732463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35592 (local address 127.0.98.1:46463)
0927 02:36:28.733018 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:28.733032 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:28.733054 (+ 22us) server_negotiation.cc:407] Connection header received
0927 02:36:28.733125 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:28.733130 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:28.733307 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:28.733440 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:36:28.733794 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:28.734558 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:28.737021 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:28.737775 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:28.737879 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:28.738006 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:28.738121 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 02:36:28.738223 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":98,"threads_started":1}
I20260927 02:36:28.738394 16248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:28.731778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36702 (local address 127.0.98.3:35341)
0927 02:36:28.732083 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:28.732095 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:28.732658 (+ 563us) server_negotiation.cc:407] Connection header received
0927 02:36:28.732870 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:28.732876 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:28.733123 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:28.733273 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:36:28.733537 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:28.734703 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:28.736286 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:28.737041 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:28.737153 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:28.737281 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:28.738193 (+ 912us) server_negotiation.cc:299] Negotiation successful
0927 02:36:28.738275 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":105,"threads_started":1}
I20260927 02:36:28.738914 16247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:28.731623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35341 (local address 127.0.0.1:36702)
0927 02:36:28.732524 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:28.732571 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:28.732732 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:28.733269 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:36:28.733274 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:36:28.733295 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:36:28.733429 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:28.733436 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:28.734827 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:28.734830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:28.736136 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:28.736146 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:28.737956 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:28.737960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:28.738042 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:28.738064 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 02:36:28.738717 (+ 653us) client_negotiation.cc:240] Negotiation successful
0927 02:36:28.738790 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"mutex_wait_us":73,"spinlock_wait_cycles":10880,"thread_start_us":68,"threads_started":1}
I20260927 02:36:28.740483 16035 tablet_service.cc:1467] Processing CreateTablet for tablet 6b36bf1a9e1840f2ae8f34910d5cd7c1 (DEFAULT_TABLE table=test-table [id=c70ebf1b16a14b5ab0cae2950abb75ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:36:28.740988 15901 tablet_service.cc:1467] Processing CreateTablet for tablet 6b36bf1a9e1840f2ae8f34910d5cd7c1 (DEFAULT_TABLE table=test-table [id=c70ebf1b16a14b5ab0cae2950abb75ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:36:28.741453 16035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b36bf1a9e1840f2ae8f34910d5cd7c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:36:28.741484 16169 tablet_service.cc:1467] Processing CreateTablet for tablet 6b36bf1a9e1840f2ae8f34910d5cd7c1 (DEFAULT_TABLE table=test-table [id=c70ebf1b16a14b5ab0cae2950abb75ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:36:28.741971 15901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b36bf1a9e1840f2ae8f34910d5cd7c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:36:28.742400 16169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b36bf1a9e1840f2ae8f34910d5cd7c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:36:28.747355 16252 tablet_bootstrap.cc:492] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P fb9f7e48bd2a42b0ad0cee0ee135f636: Bootstrap starting.
I20260927 02:36:28.748561 16253 tablet_bootstrap.cc:492] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P bcf36ad493654e22a988b3d5fa661f10: Bootstrap starting.
I20260927 02:36:28.749258 16254 tablet_bootstrap.cc:492] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P 060b505faf464a789ccc22f1bf18fa17: Bootstrap starting.
I20260927 02:36:28.750790 16252 tablet_bootstrap.cc:654] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P fb9f7e48bd2a42b0ad0cee0ee135f636: Neither blocks nor log segments found. Creating new log.
I20260927 02:36:28.750926 16253 tablet_bootstrap.cc:654] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P bcf36ad493654e22a988b3d5fa661f10: Neither blocks nor log segments found. Creating new log.
I20260927 02:36:28.751104 16254 tablet_bootstrap.cc:654] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P 060b505faf464a789ccc22f1bf18fa17: Neither blocks nor log segments found. Creating new log.
I20260927 02:36:28.751667 16253 log.cc:824] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P bcf36ad493654e22a988b3d5fa661f10: Log is configured to *not* fsync() on all Append() calls
I20260927 02:36:28.751663 16252 log.cc:824] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P fb9f7e48bd2a42b0ad0cee0ee135f636: Log is configured to *not* fsync() on all Append() calls
I20260927 02:36:28.752009 16254 log.cc:824] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P 060b505faf464a789ccc22f1bf18fa17: Log is configured to *not* fsync() on all Append() calls
I20260927 02:36:28.753489 16252 tablet_bootstrap.cc:492] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P fb9f7e48bd2a42b0ad0cee0ee135f636: No bootstrap required, opened a new log
I20260927 02:36:28.753540 16253 tablet_bootstrap.cc:492] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P bcf36ad493654e22a988b3d5fa661f10: No bootstrap required, opened a new log
I20260927 02:36:28.753664 16252 ts_tablet_manager.cc:1397] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P fb9f7e48bd2a42b0ad0cee0ee135f636: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 02:36:28.753683 16254 tablet_bootstrap.cc:492] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P 060b505faf464a789ccc22f1bf18fa17: No bootstrap required, opened a new log
I20260927 02:36:28.753724 16253 ts_tablet_manager.cc:1397] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P bcf36ad493654e22a988b3d5fa661f10: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.001s
I20260927 02:36:28.753815 16254 ts_tablet_manager.cc:1397] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P 060b505faf464a789ccc22f1bf18fa17: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 02:36:28.756693 16252 raft_consensus.cc:348] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P fb9f7e48bd2a42b0ad0cee0ee135f636 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060b505faf464a789ccc22f1bf18fa17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35341 } } peers { permanent_uuid: "fb9f7e48bd2a42b0ad0cee0ee135f636" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } } peers { permanent_uuid: "bcf36ad493654e22a988b3d5fa661f10" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46463 } }
I20260927 02:36:28.756886 16252 raft_consensus.cc:374] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P fb9f7e48bd2a42b0ad0cee0ee135f636 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:36:28.756915 16252 raft_consensus.cc:729] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P fb9f7e48bd2a42b0ad0cee0ee135f636 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb9f7e48bd2a42b0ad0cee0ee135f636, State: Initialized, Role: FOLLOWER
I20260927 02:36:28.757371 16252 consensus_queue.cc:260] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P fb9f7e48bd2a42b0ad0cee0ee135f636 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060b505faf464a789ccc22f1bf18fa17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35341 } } peers { permanent_uuid: "fb9f7e48bd2a42b0ad0cee0ee135f636" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } } peers { permanent_uuid: "bcf36ad493654e22a988b3d5fa661f10" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46463 } }
I20260927 02:36:28.757323 16253 raft_consensus.cc:348] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P bcf36ad493654e22a988b3d5fa661f10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060b505faf464a789ccc22f1bf18fa17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35341 } } peers { permanent_uuid: "fb9f7e48bd2a42b0ad0cee0ee135f636" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } } peers { permanent_uuid: "bcf36ad493654e22a988b3d5fa661f10" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46463 } }
I20260927 02:36:28.757323 16254 raft_consensus.cc:348] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P 060b505faf464a789ccc22f1bf18fa17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060b505faf464a789ccc22f1bf18fa17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35341 } } peers { permanent_uuid: "fb9f7e48bd2a42b0ad0cee0ee135f636" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } } peers { permanent_uuid: "bcf36ad493654e22a988b3d5fa661f10" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46463 } }
I20260927 02:36:28.757571 16254 raft_consensus.cc:374] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P 060b505faf464a789ccc22f1bf18fa17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:36:28.757594 16253 raft_consensus.cc:374] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P bcf36ad493654e22a988b3d5fa661f10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:36:28.757632 16254 raft_consensus.cc:729] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P 060b505faf464a789ccc22f1bf18fa17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 060b505faf464a789ccc22f1bf18fa17, State: Initialized, Role: FOLLOWER
I20260927 02:36:28.757652 16253 raft_consensus.cc:729] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P bcf36ad493654e22a988b3d5fa661f10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf36ad493654e22a988b3d5fa661f10, State: Initialized, Role: FOLLOWER
I20260927 02:36:28.758066 16253 consensus_queue.cc:260] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P bcf36ad493654e22a988b3d5fa661f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060b505faf464a789ccc22f1bf18fa17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35341 } } peers { permanent_uuid: "fb9f7e48bd2a42b0ad0cee0ee135f636" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } } peers { permanent_uuid: "bcf36ad493654e22a988b3d5fa661f10" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46463 } }
I20260927 02:36:28.758066 16254 consensus_queue.cc:260] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P 060b505faf464a789ccc22f1bf18fa17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060b505faf464a789ccc22f1bf18fa17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35341 } } peers { permanent_uuid: "fb9f7e48bd2a42b0ad0cee0ee135f636" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } } peers { permanent_uuid: "bcf36ad493654e22a988b3d5fa661f10" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46463 } }
I20260927 02:36:28.758185 16252 ts_tablet_manager.cc:1428] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P fb9f7e48bd2a42b0ad0cee0ee135f636: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260927 02:36:28.758517 16101 heartbeater.cc:503] Master 127.0.98.62:39525 was elected leader, sending a full tablet report...
I20260927 02:36:28.758622 16235 heartbeater.cc:503] Master 127.0.98.62:39525 was elected leader, sending a full tablet report...
I20260927 02:36:28.758863 15967 heartbeater.cc:503] Master 127.0.98.62:39525 was elected leader, sending a full tablet report...
I20260927 02:36:28.758917 16254 ts_tablet_manager.cc:1428] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P 060b505faf464a789ccc22f1bf18fa17: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 02:36:28.760018 16253 ts_tablet_manager.cc:1428] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P bcf36ad493654e22a988b3d5fa661f10: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
W20260927 02:36:28.795555 15968 tablet.cc:2369] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P bcf36ad493654e22a988b3d5fa661f10: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:36:28.944985 16236 tablet.cc:2369] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P 060b505faf464a789ccc22f1bf18fa17: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:36:28.969938 16258 raft_consensus.cc:482] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P fb9f7e48bd2a42b0ad0cee0ee135f636 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:36:28.970131 16258 raft_consensus.cc:504] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P fb9f7e48bd2a42b0ad0cee0ee135f636 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060b505faf464a789ccc22f1bf18fa17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35341 } } peers { permanent_uuid: "fb9f7e48bd2a42b0ad0cee0ee135f636" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } } peers { permanent_uuid: "bcf36ad493654e22a988b3d5fa661f10" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46463 } }
I20260927 02:36:28.971166 16258 leader_election.cc:288] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P fb9f7e48bd2a42b0ad0cee0ee135f636 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 060b505faf464a789ccc22f1bf18fa17 (127.0.98.3:35341), bcf36ad493654e22a988b3d5fa661f10 (127.0.98.1:46463)
I20260927 02:36:28.974627 16248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:28.971521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50091 (local address 127.0.98.3:35341)
0927 02:36:28.971659 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:28.971663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:28.971678 (+ 15us) server_negotiation.cc:407] Connection header received
0927 02:36:28.971731 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:28.971735 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:28.971790 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:28.971870 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:36:28.972126 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:28.972608 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:28.973693 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:28.974124 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:28.974228 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:28.974286 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:28.974446 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 02:36:28.974510 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 02:36:28.974627 16103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:28.971394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35341 (local address 127.0.98.2:50091)
0927 02:36:28.971526 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:28.971543 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:28.971632 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:28.971886 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:36:28.971889 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:36:28.971911 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:36:28.972029 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:28.972036 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:28.972714 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:28.972717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:28.973578 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:28.973588 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:28.974231 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:28.974234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:28.974338 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:28.974359 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 02:36:28.974424 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 02:36:28.974504 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 02:36:28.975107 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:28.971688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46463 (local address 127.0.98.2:60139)
0927 02:36:28.972089 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:28.972102 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:28.972182 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:28.972403 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:36:28.972405 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:36:28.972421 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:36:28.972519 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:28.972524 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:28.973233 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:28.973236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:28.974028 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:28.974037 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:28.974710 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:28.974714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:28.974809 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:28.974832 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 02:36:28.974905 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 02:36:28.974970 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":140,"threads_started":1}
I20260927 02:36:28.975147 16251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:28.971771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60139 (local address 127.0.98.1:46463)
0927 02:36:28.971895 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:28.971898 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:28.972160 (+ 262us) server_negotiation.cc:407] Connection header received
0927 02:36:28.972277 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:28.972280 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:28.972323 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:28.972383 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:36:28.972617 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:28.973138 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:28.974138 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:28.974578 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:28.974681 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:28.974738 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:28.974937 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 02:36:28.975034 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 02:36:28.975219 16189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b36bf1a9e1840f2ae8f34910d5cd7c1" candidate_uuid: "fb9f7e48bd2a42b0ad0cee0ee135f636" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "060b505faf464a789ccc22f1bf18fa17" is_pre_election: true
I20260927 02:36:28.975487 16189 raft_consensus.cc:2454] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P 060b505faf464a789ccc22f1bf18fa17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb9f7e48bd2a42b0ad0cee0ee135f636 in term 0.
I20260927 02:36:28.975790 15921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b36bf1a9e1840f2ae8f34910d5cd7c1" candidate_uuid: "fb9f7e48bd2a42b0ad0cee0ee135f636" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf36ad493654e22a988b3d5fa661f10" is_pre_election: true
I20260927 02:36:28.975946 15991 leader_election.cc:302] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P fb9f7e48bd2a42b0ad0cee0ee135f636 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 060b505faf464a789ccc22f1bf18fa17, fb9f7e48bd2a42b0ad0cee0ee135f636; no voters:
I20260927 02:36:28.976037 15921 raft_consensus.cc:2454] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P bcf36ad493654e22a988b3d5fa661f10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb9f7e48bd2a42b0ad0cee0ee135f636 in term 0.
I20260927 02:36:28.976200 16258 raft_consensus.cc:2781] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P fb9f7e48bd2a42b0ad0cee0ee135f636 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:36:28.976289 16258 raft_consensus.cc:482] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P fb9f7e48bd2a42b0ad0cee0ee135f636 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:36:28.976341 16258 raft_consensus.cc:3037] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P fb9f7e48bd2a42b0ad0cee0ee135f636 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:36:28.977416 16258 raft_consensus.cc:504] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P fb9f7e48bd2a42b0ad0cee0ee135f636 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060b505faf464a789ccc22f1bf18fa17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35341 } } peers { permanent_uuid: "fb9f7e48bd2a42b0ad0cee0ee135f636" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } } peers { permanent_uuid: "bcf36ad493654e22a988b3d5fa661f10" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46463 } }
I20260927 02:36:28.977736 16258 leader_election.cc:288] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P fb9f7e48bd2a42b0ad0cee0ee135f636 [CANDIDATE]: Term 1 election: Requested vote from peers 060b505faf464a789ccc22f1bf18fa17 (127.0.98.3:35341), bcf36ad493654e22a988b3d5fa661f10 (127.0.98.1:46463)
I20260927 02:36:28.978034 16189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b36bf1a9e1840f2ae8f34910d5cd7c1" candidate_uuid: "fb9f7e48bd2a42b0ad0cee0ee135f636" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "060b505faf464a789ccc22f1bf18fa17"
I20260927 02:36:28.978140 15921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b36bf1a9e1840f2ae8f34910d5cd7c1" candidate_uuid: "fb9f7e48bd2a42b0ad0cee0ee135f636" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf36ad493654e22a988b3d5fa661f10"
I20260927 02:36:28.978155 16189 raft_consensus.cc:3037] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P 060b505faf464a789ccc22f1bf18fa17 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:36:28.978251 15921 raft_consensus.cc:3037] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P bcf36ad493654e22a988b3d5fa661f10 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:36:28.979350 15921 raft_consensus.cc:2454] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P bcf36ad493654e22a988b3d5fa661f10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb9f7e48bd2a42b0ad0cee0ee135f636 in term 1.
I20260927 02:36:28.979350 16189 raft_consensus.cc:2454] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P 060b505faf464a789ccc22f1bf18fa17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb9f7e48bd2a42b0ad0cee0ee135f636 in term 1.
I20260927 02:36:28.979637 15991 leader_election.cc:302] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P fb9f7e48bd2a42b0ad0cee0ee135f636 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 060b505faf464a789ccc22f1bf18fa17, fb9f7e48bd2a42b0ad0cee0ee135f636; no voters:
I20260927 02:36:28.979815 16258 raft_consensus.cc:2781] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P fb9f7e48bd2a42b0ad0cee0ee135f636 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:36:28.979982 16258 raft_consensus.cc:686] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P fb9f7e48bd2a42b0ad0cee0ee135f636 [term 1 LEADER]: Becoming Leader. State: Replica: fb9f7e48bd2a42b0ad0cee0ee135f636, State: Running, Role: LEADER
I20260927 02:36:28.980257 16258 consensus_queue.cc:237] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P fb9f7e48bd2a42b0ad0cee0ee135f636 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060b505faf464a789ccc22f1bf18fa17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35341 } } peers { permanent_uuid: "fb9f7e48bd2a42b0ad0cee0ee135f636" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } } peers { permanent_uuid: "bcf36ad493654e22a988b3d5fa661f10" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46463 } }
I20260927 02:36:28.983073 15774 catalog_manager.cc:5495] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P fb9f7e48bd2a42b0ad0cee0ee135f636 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb9f7e48bd2a42b0ad0cee0ee135f636 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "fb9f7e48bd2a42b0ad0cee0ee135f636" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060b505faf464a789ccc22f1bf18fa17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb9f7e48bd2a42b0ad0cee0ee135f636" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf36ad493654e22a988b3d5fa661f10" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46463 } health_report { overall_health: UNKNOWN } } }
W20260927 02:36:29.002555 16102 tablet.cc:2369] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P fb9f7e48bd2a42b0ad0cee0ee135f636: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:36:29.032193 16250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:29.028920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54064 (local address 127.0.98.2:40749)
0927 02:36:29.029063 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:29.029068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:29.029082 (+ 14us) server_negotiation.cc:407] Connection header received
0927 02:36:29.029195 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:29.029198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:29.029251 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:29.029338 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:36:29.029566 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:29.030048 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:29.030871 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:29.031076 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:29.031168 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:29.031435 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:36:29.031820 (+ 385us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:36:29.031924 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:29.032002 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 02:36:29.032054 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 02:36:29.037413 16189 raft_consensus.cc:1261] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P 060b505faf464a789ccc22f1bf18fa17 [term 1 FOLLOWER]: Refusing update from remote peer fb9f7e48bd2a42b0ad0cee0ee135f636: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:36:29.037451 15921 raft_consensus.cc:1261] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P bcf36ad493654e22a988b3d5fa661f10 [term 1 FOLLOWER]: Refusing update from remote peer fb9f7e48bd2a42b0ad0cee0ee135f636: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:36:29.038048 16262 consensus_queue.cc:1035] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P fb9f7e48bd2a42b0ad0cee0ee135f636 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf36ad493654e22a988b3d5fa661f10" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:36:29.038209 16258 consensus_queue.cc:1035] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P fb9f7e48bd2a42b0ad0cee0ee135f636 [LEADER]: Connected to new peer: Peer: permanent_uuid: "060b505faf464a789ccc22f1bf18fa17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:36:29.046219 16267 mvcc.cc:205] Tried to move back new op lower bound from 7333792108690784256 to 7333792108465258496. Current Snapshot: MvccSnapshot[applied={T|T < 7333792108690784256}]
I20260927 02:36:29.051698 15774 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:36:29.051831 15774 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:36:29.053874 15774 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836: Sending DeleteTablet for 3 replicas of tablet 6b36bf1a9e1840f2ae8f34910d5cd7c1
I20260927 02:36:29.054728 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 15838
I20260927 02:36:29.054719 16169 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b36bf1a9e1840f2ae8f34910d5cd7c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:36:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36702
I20260927 02:36:29.055161 16035 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b36bf1a9e1840f2ae8f34910d5cd7c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:36:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54056
I20260927 02:36:29.055554 16275 tablet_replica.cc:330] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P fb9f7e48bd2a42b0ad0cee0ee135f636: stopping tablet replica
I20260927 02:36:29.055866 16275 raft_consensus.cc:2229] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P fb9f7e48bd2a42b0ad0cee0ee135f636 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:36:29.056030 16274 tablet_replica.cc:330] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P 060b505faf464a789ccc22f1bf18fa17: stopping tablet replica
I20260927 02:36:29.056296 16274 raft_consensus.cc:2229] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P 060b505faf464a789ccc22f1bf18fa17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:36:29.056321 16275 raft_consensus.cc:2258] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P fb9f7e48bd2a42b0ad0cee0ee135f636 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:36:29.056550 16274 raft_consensus.cc:2258] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P 060b505faf464a789ccc22f1bf18fa17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:36:29.057875 16274 ts_tablet_manager.cc:1905] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P 060b505faf464a789ccc22f1bf18fa17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:36:29.057874 16275 ts_tablet_manager.cc:1905] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P fb9f7e48bd2a42b0ad0cee0ee135f636: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:36:29.060355 16274 ts_tablet_manager.cc:1918] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P 060b505faf464a789ccc22f1bf18fa17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:36:29.060443 16274 log.cc:1196] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P 060b505faf464a789ccc22f1bf18fa17: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-2/wal/wals/6b36bf1a9e1840f2ae8f34910d5cd7c1
I20260927 02:36:29.060627 16275 ts_tablet_manager.cc:1918] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P fb9f7e48bd2a42b0ad0cee0ee135f636: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:36:29.060725 16274 ts_tablet_manager.cc:1939] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P 060b505faf464a789ccc22f1bf18fa17: Deleting consensus metadata
I20260927 02:36:29.060833 16275 log.cc:1196] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P fb9f7e48bd2a42b0ad0cee0ee135f636: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-1/wal/wals/6b36bf1a9e1840f2ae8f34910d5cd7c1
I20260927 02:36:29.061205 16275 ts_tablet_manager.cc:1939] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P fb9f7e48bd2a42b0ad0cee0ee135f636: Deleting consensus metadata
W20260927 02:36:29.061793 15763 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46463 (error 108)
I20260927 02:36:29.062295 15763 catalog_manager.cc:4841] TS 060b505faf464a789ccc22f1bf18fa17 (127.0.98.3:35341): tablet 6b36bf1a9e1840f2ae8f34910d5cd7c1 (table test-table [id=c70ebf1b16a14b5ab0cae2950abb75ce]) successfully deleted
I20260927 02:36:29.062392 15762 catalog_manager.cc:4841] TS fb9f7e48bd2a42b0ad0cee0ee135f636 (127.0.98.2:40749): tablet 6b36bf1a9e1840f2ae8f34910d5cd7c1 (table test-table [id=c70ebf1b16a14b5ab0cae2950abb75ce]) successfully deleted
I20260927 02:36:29.062817 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 15972
I20260927 02:36:29.065753 16247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:29.065436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46463 (local address 127.0.0.1:35600)
0927 02:36:29.065588 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:29.065663 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 02:36:29.065908 15763 catalog_manager.cc:4584] TS bcf36ad493654e22a988b3d5fa661f10 (127.0.98.1:46463): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6b36bf1a9e1840f2ae8f34910d5cd7c1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46463: connect: Connection refused (error 111)
I20260927 02:36:29.068261 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 16106
I20260927 02:36:29.072625 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 15744
I20260927 02:36:49.077934 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39525
--webserver_interface=127.0.98.62
--webserver_port=35165
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39525
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:36:49.182103 16277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:36:49.182327 16277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:36:49.182375 16277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:36:49.186208 16277 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 02:36:49.186247 16277 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 02:36:49.186267 16277 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 02:36:49.186285 16277 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:36:49.186307 16277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:36:49.186336 16277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:36:49.186365 16277 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:36:49.186388 16277 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:36:49.191291 16277 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:36:49.192159 16277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:36:49.193051 16277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:36:49.204694 16283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:36:49.204674 16286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:36:49.204754 16284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:36:49.204824 16277 server_base.cc:1034] running on GCE node
I20260927 02:36:49.205451 16277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:36:49.206465 16277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:36:49.207612 16277 hybrid_clock.cc:648] HybridClock initialized: now 1790476609207597 us; error 29 us; skew 500 ppm
Sep 27 02:36:49 dist-test-slave-grlk krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476609, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:36:49.210162 16277 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:36:49.210983 16277 webserver.cc:460] Webserver started at http://127.0.98.62:35165/ using document root <none> and password file <none>
I20260927 02:36:49.211524 16277 fs_manager.cc:359] Metadata directory not provided
I20260927 02:36:49.211598 16277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:36:49.215081 16277 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260927 02:36:49.217150 16293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:49.218400 16277 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:36:49.218516 16277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "4f19bae86c5b42b0b6a725e5b1b20836"
format_stamp: "Formatted at 2026-09-27 02:36:27 on dist-test-slave-grlk"
server_key: "b0699c1d4624bea5a4f5c11aa99316e2"
server_key_iv: "2fa3876119f1a2b575804cc2a56b3967"
server_key_version: "encryptionkey@0"
I20260927 02:36:49.219086 16277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:36:49.235236 16277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:36:49.237846 16277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:36:49.238003 16277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:36:49.244320 16277 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39525
I20260927 02:36:49.244570 16345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39525 every 8 connection(s)
I20260927 02:36:49.245396 16277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:36:49.249761 16346 sys_catalog.cc:263] Verifying existing consensus state
I20260927 02:36:49.251279 16346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836: Bootstrap starting.
I20260927 02:36:49.253283 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 16277
I20260927 02:36:49.253641 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46463
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39525
--webserver_port=45407
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:36:49.267210 16346 log.cc:824] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836: Log is configured to *not* fsync() on all Append() calls
I20260927 02:36:49.275672 16346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:36:49.276048 16346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836: Bootstrap complete.
I20260927 02:36:49.279870 16346 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f19bae86c5b42b0b6a725e5b1b20836" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39525 } }
I20260927 02:36:49.280251 16346 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f19bae86c5b42b0b6a725e5b1b20836, State: Initialized, Role: FOLLOWER
I20260927 02:36:49.280683 16346 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f19bae86c5b42b0b6a725e5b1b20836" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39525 } }
I20260927 02:36:49.280810 16346 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:36:49.280879 16346 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:36:49.280987 16346 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [term 1 FOLLOWER]: Advancing to term 2
I20260927 02:36:49.282752 16346 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f19bae86c5b42b0b6a725e5b1b20836" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39525 } }
I20260927 02:36:49.283044 16346 leader_election.cc:302] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f19bae86c5b42b0b6a725e5b1b20836; no voters:
I20260927 02:36:49.283324 16346 leader_election.cc:288] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 02:36:49.283438 16350 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 02:36:49.283643 16350 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [term 2 LEADER]: Becoming Leader. State: Replica: 4f19bae86c5b42b0b6a725e5b1b20836, State: Running, Role: LEADER
I20260927 02:36:49.283974 16350 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f19bae86c5b42b0b6a725e5b1b20836" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39525 } }
I20260927 02:36:49.284484 16346 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:36:49.285399 16351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4f19bae86c5b42b0b6a725e5b1b20836" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f19bae86c5b42b0b6a725e5b1b20836" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39525 } } }
I20260927 02:36:49.285406 16352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f19bae86c5b42b0b6a725e5b1b20836. Latest consensus state: current_term: 2 leader_uuid: "4f19bae86c5b42b0b6a725e5b1b20836" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f19bae86c5b42b0b6a725e5b1b20836" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39525 } } }
I20260927 02:36:49.285532 16351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [sys.catalog]: This master's current role is: LEADER
I20260927 02:36:49.285549 16352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [sys.catalog]: This master's current role is: LEADER
I20260927 02:36:49.285908 16359 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:36:49.290341 16359 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:36:49.291145 16359 catalog_manager.cc:1259] Loaded cluster ID: 6620759ac746484aa0f68da98198dbed
I20260927 02:36:49.291204 16359 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:36:49.293687 16359 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:36:49.294363 16359 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836: Loaded TSK: 0
I20260927 02:36:49.309407 16359 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836: Generated new TSK 1
I20260927 02:36:49.309612 16359 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 02:36:49.359390 16348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:36:49.359632 16348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:36:49.359687 16348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:36:49.363691 16348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:36:49.363750 16348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:36:49.363826 16348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:36:49.369459 16348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46463
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=45407
--tserver_master_addrs=127.0.98.62:39525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:36:49.370491 16348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:36:49.371577 16348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:36:49.381806 16374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:36:49.381807 16373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:36:49.382068 16348 server_base.cc:1034] running on GCE node
W20260927 02:36:49.382138 16376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:36:49.382568 16348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:36:49.383314 16348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:36:49.384507 16348 hybrid_clock.cc:648] HybridClock initialized: now 1790476609384478 us; error 39 us; skew 500 ppm
Sep 27 02:36:49 dist-test-slave-grlk krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476609, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:36:49.387796 16348 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:36:49.388860 16348 webserver.cc:460] Webserver started at http://127.0.98.1:45407/ using document root <none> and password file <none>
I20260927 02:36:49.389544 16348 fs_manager.cc:359] Metadata directory not provided
I20260927 02:36:49.389650 16348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:36:49.394747 16348 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:36:49.397346 16383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:49.398751 16348 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:36:49.398873 16348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-0/wal
uuid: "bcf36ad493654e22a988b3d5fa661f10"
format_stamp: "Formatted at 2026-09-27 02:36:28 on dist-test-slave-grlk"
server_key: "bd1dae9cfba5669d741e56ce9325ef77"
server_key_iv: "845aaf2fe601841a23a95204480fecfd"
server_key_version: "encryptionkey@0"
I20260927 02:36:49.399556 16348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:36:49.417565 16348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:36:49.421064 16348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:36:49.421243 16348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:36:49.421880 16348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:36:49.423115 16390 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 02:36:49.426981 16348 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 02:36:49.427058 16348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260927 02:36:49.427119 16348 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 02:36:49.429298 16348 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 02:36:49.429354 16348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260927 02:36:49.429656 16390 tablet_bootstrap.cc:492] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P bcf36ad493654e22a988b3d5fa661f10: Bootstrap starting.
I20260927 02:36:49.439181 16348 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46463
I20260927 02:36:49.439438 16497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46463 every 8 connection(s)
I20260927 02:36:49.440332 16348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:36:49.449241 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 16348
I20260927 02:36:49.449781 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40749
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39525
--webserver_port=33353
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:36:49.450203 16390 log.cc:824] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P bcf36ad493654e22a988b3d5fa661f10: Log is configured to *not* fsync() on all Append() calls
Sep 27 02:36:49 dist-test-slave-grlk krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476609, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:36:49.457527 16390 tablet_bootstrap.cc:492] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P bcf36ad493654e22a988b3d5fa661f10: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:36:49.457964 16390 tablet_bootstrap.cc:492] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P bcf36ad493654e22a988b3d5fa661f10: Bootstrap complete.
I20260927 02:36:49.458297 16501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:49.442580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40517 (local address 127.0.98.62:39525)
0927 02:36:49.443137 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:49.443157 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:49.443211 (+ 54us) server_negotiation.cc:407] Connection header received
0927 02:36:49.444319 (+ 1108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:49.444335 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:49.444767 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:49.445208 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:36:49.446213 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:49.447882 (+ 1669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:49.449000 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:49.449670 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:49.449834 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:49.452587 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:36:49.452594 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:36:49.452609 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:36:49.452643 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:36:49.454763 (+ 2120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:49.455379 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:49.455385 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:49.455391 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:49.455561 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:49.455931 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:49.455934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:49.455936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:49.456392 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:36:49.456543 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:49.457135 (+ 592us) server_negotiation.cc:299] Negotiation successful
0927 02:36:49.457431 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":132,"threads_started":1}
I20260927 02:36:49.458315 16500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:49.442501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.1:40517)
0927 02:36:49.442974 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:49.443026 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:49.443832 (+ 806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:49.445239 (+ 1407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:36:49.445246 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:36:49.445580 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:36:49.446035 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:49.446049 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:49.448004 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:49.448006 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:49.448852 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:49.448860 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:49.449804 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:49.449807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:49.449977 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:49.450555 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:36:49.450572 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:36:49.452380 (+ 1808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:36:49.454907 (+ 2527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:49.454913 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:49.454926 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:49.455262 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:49.455679 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:49.455682 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:49.455684 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:49.455835 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:49.456528 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:36:49.456533 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:36:49.456801 (+ 268us) client_negotiation.cc:769] Sending connection context
0927 02:36:49.457007 (+ 206us) client_negotiation.cc:240] Negotiation successful
0927 02:36:49.457327 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":112,"threads_started":1}
I20260927 02:36:49.459213 16390 ts_tablet_manager.cc:1397] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P bcf36ad493654e22a988b3d5fa661f10: Time spent bootstrapping tablet: real 0.030s user 0.014s sys 0.013s
I20260927 02:36:49.461264 16498 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39525
I20260927 02:36:49.461517 16498 heartbeater.cc:464] Registering TS with master...
I20260927 02:36:49.462061 16498 heartbeater.cc:511] Master 127.0.98.62:39525 requested a full tablet report, sending...
I20260927 02:36:49.461788 16390 raft_consensus.cc:348] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P bcf36ad493654e22a988b3d5fa661f10 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060b505faf464a789ccc22f1bf18fa17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35341 } } peers { permanent_uuid: "fb9f7e48bd2a42b0ad0cee0ee135f636" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } } peers { permanent_uuid: "bcf36ad493654e22a988b3d5fa661f10" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46463 } }
I20260927 02:36:49.462216 16390 raft_consensus.cc:729] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P bcf36ad493654e22a988b3d5fa661f10 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf36ad493654e22a988b3d5fa661f10, State: Initialized, Role: FOLLOWER
I20260927 02:36:49.462662 16390 consensus_queue.cc:260] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P bcf36ad493654e22a988b3d5fa661f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060b505faf464a789ccc22f1bf18fa17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35341 } } peers { permanent_uuid: "fb9f7e48bd2a42b0ad0cee0ee135f636" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } } peers { permanent_uuid: "bcf36ad493654e22a988b3d5fa661f10" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46463 } }
I20260927 02:36:49.464479 16390 ts_tablet_manager.cc:1428] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P bcf36ad493654e22a988b3d5fa661f10: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 02:36:49.465066 16310 ts_manager.cc:194] Registered new tserver with Master: bcf36ad493654e22a988b3d5fa661f10 (127.0.98.1:46463)
I20260927 02:36:49.468133 16310 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40517
I20260927 02:36:49.470535 16498 heartbeater.cc:503] Master 127.0.98.62:39525 was elected leader, sending a full tablet report...
Sep 27 02:36:49 dist-test-slave-grlk krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476609, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:36:49.477545 16506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:49.467750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48516 (local address 127.0.98.1:46463)
0927 02:36:49.468109 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:49.468120 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:49.468511 (+ 391us) server_negotiation.cc:407] Connection header received
0927 02:36:49.468680 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:49.468686 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:49.468889 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:49.469070 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:36:49.469752 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:49.470889 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:49.471738 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:49.472175 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:49.472266 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:49.474484 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:36:49.474492 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:36:49.474500 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:36:49.474521 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:36:49.475869 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:49.476359 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:49.476363 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:49.476364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:49.476435 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:49.476681 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:49.476683 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:49.476684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:49.476877 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:36:49.477007 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:49.477315 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 02:36:49.477432 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":100,"threads_started":1}
I20260927 02:36:49.477586 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:49.467664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46463 (local address 127.0.0.1:48516)
0927 02:36:49.468362 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:49.468402 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:49.468535 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:49.469069 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:36:49.469075 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:36:49.469501 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:36:49.469633 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:49.469643 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:49.470990 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:49.470993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:49.471629 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:49.471639 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:49.472267 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:49.472282 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:49.472387 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:49.472860 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:36:49.472872 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:36:49.474360 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:36:49.475968 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:49.475977 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:49.475982 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:49.476265 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:49.476510 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:49.476513 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:49.476515 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:49.476605 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:49.476989 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:36:49.477002 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:36:49.477118 (+ 116us) client_negotiation.cc:769] Sending connection context
0927 02:36:49.477257 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 02:36:49.477425 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":117,"threads_started":1}
I20260927 02:36:49.478523 16432 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b36bf1a9e1840f2ae8f34910d5cd7c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:36:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48516
I20260927 02:36:49.479447 16507 tablet_replica.cc:330] stopping tablet replica
I20260927 02:36:49.479689 16507 raft_consensus.cc:2229] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P bcf36ad493654e22a988b3d5fa661f10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:36:49.479894 16507 raft_consensus.cc:2258] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P bcf36ad493654e22a988b3d5fa661f10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:36:49.480748 16507 ts_tablet_manager.cc:1905] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P bcf36ad493654e22a988b3d5fa661f10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:36:49.484004 16507 ts_tablet_manager.cc:1918] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P bcf36ad493654e22a988b3d5fa661f10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:36:49.484086 16507 log.cc:1196] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P bcf36ad493654e22a988b3d5fa661f10: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-0/wal/wals/6b36bf1a9e1840f2ae8f34910d5cd7c1
I20260927 02:36:49.484328 16507 ts_tablet_manager.cc:1939] T 6b36bf1a9e1840f2ae8f34910d5cd7c1 P bcf36ad493654e22a988b3d5fa661f10: Deleting consensus metadata
I20260927 02:36:49.486159 16297 catalog_manager.cc:4841] TS bcf36ad493654e22a988b3d5fa661f10 (127.0.98.1:46463): tablet 6b36bf1a9e1840f2ae8f34910d5cd7c1 (table test-table [id=c70ebf1b16a14b5ab0cae2950abb75ce]) successfully deleted
W20260927 02:36:49.561625 16503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:36:49.561867 16503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:36:49.561908 16503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:36:49.565965 16503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:36:49.566018 16503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:36:49.566085 16503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:36:49.571092 16503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40749
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33353
--tserver_master_addrs=127.0.98.62:39525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:36:49.571979 16503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:36:49.572878 16503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:36:49.581699 16512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:36:49.581765 16503 server_base.cc:1034] running on GCE node
W20260927 02:36:49.581707 16513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:36:49.581684 16515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:36:49.582262 16503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:36:49.582808 16503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:36:49.583972 16503 hybrid_clock.cc:648] HybridClock initialized: now 1790476609583956 us; error 24 us; skew 500 ppm
Sep 27 02:36:49 dist-test-slave-grlk krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476609, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:36:49.586541 16503 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:36:49.587275 16503 webserver.cc:460] Webserver started at http://127.0.98.2:33353/ using document root <none> and password file <none>
I20260927 02:36:49.587741 16503 fs_manager.cc:359] Metadata directory not provided
I20260927 02:36:49.587801 16503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:36:49.591163 16503 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.001s
I20260927 02:36:49.593089 16522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:49.594180 16503 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:36:49.594278 16503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-1/wal
uuid: "fb9f7e48bd2a42b0ad0cee0ee135f636"
format_stamp: "Formatted at 2026-09-27 02:36:28 on dist-test-slave-grlk"
server_key: "b572203f42735d94f070734bd05f9f55"
server_key_iv: "4767fea16b5ad0c79da83fc0524a3d1f"
server_key_version: "encryptionkey@0"
I20260927 02:36:49.594754 16503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:36:49.612115 16503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:36:49.614568 16503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:36:49.614698 16503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:36:49.615156 16503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:36:49.615869 16503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:36:49.615911 16503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:49.615947 16503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:36:49.615967 16503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:49.624006 16503 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40749
I20260927 02:36:49.624050 16635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40749 every 8 connection(s)
I20260927 02:36:49.624907 16503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:36:49.626389 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 16503
I20260927 02:36:49.626812 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35341
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39525
--webserver_port=36991
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 02:36:49 dist-test-slave-grlk krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476609, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:36:49.637938 16501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:49.626567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53711 (local address 127.0.98.62:39525)
0927 02:36:49.626735 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:49.626739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:49.627632 (+ 893us) server_negotiation.cc:407] Connection header received
0927 02:36:49.628991 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:49.628994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:49.629055 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:49.629149 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:36:49.630294 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:49.630829 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:49.631947 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:49.632188 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:49.632276 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:49.634722 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:36:49.634725 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:36:49.634727 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:36:49.634754 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:36:49.636038 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:49.636501 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:49.636504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:49.636505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:49.636577 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:49.636875 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:49.636879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:49.636880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:49.637087 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:36:49.637185 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:49.637641 (+ 456us) server_negotiation.cc:299] Negotiation successful
0927 02:36:49.637771 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 02:36:49.638702 16638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:49.627009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.2:53711)
0927 02:36:49.627466 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:49.627520 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:49.628773 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:49.629319 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:36:49.629326 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:36:49.629676 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:36:49.630140 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:49.630154 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:49.630951 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:49.630954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:49.631827 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:49.631836 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:49.632324 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:49.632328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:49.632479 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:49.633150 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:36:49.633167 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:36:49.634546 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:36:49.636145 (+ 1599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:49.636151 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:49.636166 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:49.636409 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:49.636659 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:49.636662 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:49.636664 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:49.636793 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:49.637192 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:36:49.637198 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:36:49.637425 (+ 227us) client_negotiation.cc:769] Sending connection context
0927 02:36:49.637583 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 02:36:49.637905 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":115,"threads_started":1}
I20260927 02:36:49.639756 16636 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39525
I20260927 02:36:49.639974 16636 heartbeater.cc:464] Registering TS with master...
I20260927 02:36:49.640424 16636 heartbeater.cc:511] Master 127.0.98.62:39525 requested a full tablet report, sending...
I20260927 02:36:49.641417 16310 ts_manager.cc:194] Registered new tserver with Master: fb9f7e48bd2a42b0ad0cee0ee135f636 (127.0.98.2:40749)
I20260927 02:36:49.642135 16310 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53711
W20260927 02:36:49.731763 16639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:36:49.732002 16639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:36:49.732043 16639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:36:49.735891 16639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:36:49.735939 16639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:36:49.735993 16639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:36:49.740876 16639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35341
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36991
--tserver_master_addrs=127.0.98.62:39525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:36:49.741739 16639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:36:49.742664 16639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:36:49.751468 16645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:36:49.751550 16644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:36:49.751485 16639 server_base.cc:1034] running on GCE node
W20260927 02:36:49.751513 16647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:36:49.752063 16639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:36:49.752619 16639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:36:49.753767 16639 hybrid_clock.cc:648] HybridClock initialized: now 1790476609753751 us; error 33 us; skew 500 ppm
Sep 27 02:36:49 dist-test-slave-grlk krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476609, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:36:49.756310 16639 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:36:49.757180 16639 webserver.cc:460] Webserver started at http://127.0.98.3:36991/ using document root <none> and password file <none>
I20260927 02:36:49.757705 16639 fs_manager.cc:359] Metadata directory not provided
I20260927 02:36:49.757777 16639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:36:49.761323 16639 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:36:49.763224 16654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:49.764328 16639 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:36:49.764420 16639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-2/wal
uuid: "060b505faf464a789ccc22f1bf18fa17"
format_stamp: "Formatted at 2026-09-27 02:36:28 on dist-test-slave-grlk"
server_key: "b1f068eccd152be1f1018a26a8a75225"
server_key_iv: "299479cda5846f2ea95e8cbdacd13760"
server_key_version: "encryptionkey@0"
I20260927 02:36:49.764958 16639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:36:49.775019 16639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:36:49.778101 16639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:36:49.778249 16639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:36:49.778774 16639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:36:49.779543 16639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:36:49.779588 16639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:49.779628 16639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:36:49.779644 16639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:36:49.787953 16639 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35341
I20260927 02:36:49.787976 16767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35341 every 8 connection(s)
I20260927 02:36:49.788846 16639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:36:49.792632 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 16639
Sep 27 02:36:49 dist-test-slave-grlk krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476609, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:36:49.802021 16772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:49.793254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38298 (local address 127.0.98.62:39525)
0927 02:36:49.793478 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:49.793481 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:49.793556 (+ 75us) server_negotiation.cc:407] Connection header received
0927 02:36:49.793680 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:49.793682 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:49.793717 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:49.793786 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:36:49.794529 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:49.795058 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:49.796141 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:49.796429 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:49.796542 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:49.798073 (+ 1531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:36:49.798076 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:36:49.798078 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:36:49.798100 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:36:49.800268 (+ 2168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:49.800677 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:49.800681 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:49.800682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:49.800759 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:49.801003 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:49.801007 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:49.801009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:49.801177 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:36:49.801355 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:49.801806 (+ 451us) server_negotiation.cc:299] Negotiation successful
0927 02:36:49.801894 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":72,"threads_started":1}
I20260927 02:36:49.802462 16501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:49.790446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35249 (local address 127.0.98.62:39525)
0927 02:36:49.790558 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:49.790561 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:49.791322 (+ 761us) server_negotiation.cc:407] Connection header received
0927 02:36:49.792557 (+ 1235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:49.792560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:49.792608 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:49.792680 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:36:49.794041 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:49.794599 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:49.796013 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:49.796286 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:49.796382 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:49.799038 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:36:49.799042 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:36:49.799044 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:36:49.799083 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:36:49.800557 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:49.801020 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:49.801023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:49.801025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:49.801100 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:49.801452 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:49.801454 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:49.801456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:49.801654 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:36:49.801744 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:49.802254 (+ 510us) server_negotiation.cc:299] Negotiation successful
0927 02:36:49.802355 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 02:36:49.803426 16770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:49.790793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.3:35249)
0927 02:36:49.791192 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:49.791229 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:49.792334 (+ 1105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:49.792867 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:36:49.792875 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:36:49.793287 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:36:49.793902 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:49.793922 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:49.794714 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:49.794717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:49.795892 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:49.795903 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:49.796435 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:49.796438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:49.796714 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:49.797329 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:36:49.797349 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:36:49.798880 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:36:49.800648 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:49.800654 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:49.800667 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:49.800891 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:49.801178 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:36:49.801181 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:36:49.801183 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:36:49.801319 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:36:49.801752 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:36:49.801759 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:36:49.801999 (+ 240us) client_negotiation.cc:769] Sending connection context
0927 02:36:49.802193 (+ 194us) client_negotiation.cc:240] Negotiation successful
0927 02:36:49.802518 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":107,"threads_started":1}
I20260927 02:36:49.804474 16768 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39525
I20260927 02:36:49.804693 16768 heartbeater.cc:464] Registering TS with master...
I20260927 02:36:49.805223 16768 heartbeater.cc:511] Master 127.0.98.62:39525 requested a full tablet report, sending...
I20260927 02:36:49.806254 16310 ts_manager.cc:194] Registered new tserver with Master: 060b505faf464a789ccc22f1bf18fa17 (127.0.98.3:35341)
I20260927 02:36:49.806775 16310 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35249
I20260927 02:36:49.815171 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:36:49.818747 16501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:49.815690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38312 (local address 127.0.98.62:39525)
0927 02:36:49.815778 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:49.815782 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:49.815959 (+ 177us) server_negotiation.cc:407] Connection header received
0927 02:36:49.816085 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:49.816087 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:49.816175 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:49.816258 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:36:49.816406 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:49.816888 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:49.817782 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:49.817999 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:49.818083 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:49.818309 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:36:49.818435 (+ 126us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 02:36:49.818652 (+ 217us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38312: token expired
Metrics: {"server-negotiator.queue_time_us":26}
W20260927 02:36:49.818795 16501 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38312: token expired
W20260927 02:36:49.819111 16238 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:39525)
I20260927 02:36:49.825625 16501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:49.819430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38324 (local address 127.0.98.62:39525)
0927 02:36:49.819538 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:49.819542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:49.819554 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:36:49.819638 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:49.819640 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:49.819678 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:49.819734 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:36:49.820289 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:49.820754 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:49.821518 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:49.821743 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:49.821823 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:49.823016 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:36:49.823020 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:36:49.823023 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:36:49.823047 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:36:49.824338 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:49.824728 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:49.824731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:49.824732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:49.824810 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:36:49.825054 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:36:49.825056 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:36:49.825057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:36:49.825213 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:36:49.825306 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:49.825442 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 02:36:49.825521 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 02:36:49.827721 16310 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:36:49.830848 16310 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:36:49.838250 16432 tablet_service.cc:1467] Processing CreateTablet for tablet 711abfa18eee412a84905888b8994321 (DEFAULT_TABLE table=test-table [id=db26bf9b29ba4d6c989107fce86c6e6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:36:49.838910 16432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 711abfa18eee412a84905888b8994321. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:36:49.841459 16390 tablet_bootstrap.cc:492] T 711abfa18eee412a84905888b8994321 P bcf36ad493654e22a988b3d5fa661f10: Bootstrap starting.
I20260927 02:36:49.841881 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:49.836771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40749 (local address 127.0.0.1:38216)
0927 02:36:49.837088 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:49.837104 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:49.837179 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:49.837910 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:36:49.837913 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:36:49.837932 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:36:49.838029 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:49.838035 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:49.839275 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:49.839279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:49.840254 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:49.840264 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:49.841337 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:49.841341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:49.841548 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:49.841593 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 02:36:49.841672 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 02:36:49.841748 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":56}
I20260927 02:36:49.842305 16390 tablet_bootstrap.cc:654] T 711abfa18eee412a84905888b8994321 P bcf36ad493654e22a988b3d5fa661f10: Neither blocks nor log segments found. Creating new log.
I20260927 02:36:49.842067 16777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:49.836979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38216 (local address 127.0.98.2:40749)
0927 02:36:49.837325 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:49.837449 (+ 124us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:49.837482 (+ 33us) server_negotiation.cc:407] Connection header received
0927 02:36:49.837562 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:49.837566 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:49.837739 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:49.837927 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:36:49.838174 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:49.839181 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:49.840384 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:49.841207 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:49.841307 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:49.841441 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:49.841704 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 02:36:49.841802 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":135,"threads_started":1}
I20260927 02:36:49.843205 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:49.837390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35341 (local address 127.0.0.1:43804)
0927 02:36:49.838091 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:49.838103 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:49.838169 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:49.838701 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:36:49.838703 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:36:49.838722 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:36:49.838807 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:49.838813 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:49.840331 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:49.840334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:49.841543 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:49.841552 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:49.842850 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:49.842853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:49.842937 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:49.842957 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 02:36:49.843019 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 02:36:49.843081 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"mutex_wait_us":65,"thread_start_us":60,"threads_started":1}
I20260927 02:36:49.843566 16779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:49.837749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43804 (local address 127.0.98.3:35341)
0927 02:36:49.838258 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:49.838275 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:49.838297 (+ 22us) server_negotiation.cc:407] Connection header received
0927 02:36:49.838366 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:49.838372 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:49.838553 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:49.838704 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:36:49.838897 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:49.840223 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:49.841636 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:49.842562 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:49.842656 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:49.842793 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:49.843354 (+ 561us) server_negotiation.cc:299] Negotiation successful
0927 02:36:49.843438 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":116,"threads_started":1}
I20260927 02:36:49.843771 16390 tablet_bootstrap.cc:492] T 711abfa18eee412a84905888b8994321 P bcf36ad493654e22a988b3d5fa661f10: No bootstrap required, opened a new log
I20260927 02:36:49.843851 16390 ts_tablet_manager.cc:1397] T 711abfa18eee412a84905888b8994321 P bcf36ad493654e22a988b3d5fa661f10: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260927 02:36:49.844215 16390 raft_consensus.cc:348] T 711abfa18eee412a84905888b8994321 P bcf36ad493654e22a988b3d5fa661f10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9f7e48bd2a42b0ad0cee0ee135f636" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } } peers { permanent_uuid: "060b505faf464a789ccc22f1bf18fa17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35341 } } peers { permanent_uuid: "bcf36ad493654e22a988b3d5fa661f10" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46463 } }
I20260927 02:36:49.844338 16390 raft_consensus.cc:374] T 711abfa18eee412a84905888b8994321 P bcf36ad493654e22a988b3d5fa661f10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:36:49.844383 16390 raft_consensus.cc:729] T 711abfa18eee412a84905888b8994321 P bcf36ad493654e22a988b3d5fa661f10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf36ad493654e22a988b3d5fa661f10, State: Initialized, Role: FOLLOWER
I20260927 02:36:49.844503 16390 consensus_queue.cc:260] T 711abfa18eee412a84905888b8994321 P bcf36ad493654e22a988b3d5fa661f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9f7e48bd2a42b0ad0cee0ee135f636" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } } peers { permanent_uuid: "060b505faf464a789ccc22f1bf18fa17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35341 } } peers { permanent_uuid: "bcf36ad493654e22a988b3d5fa661f10" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46463 } }
I20260927 02:36:49.844765 16390 ts_tablet_manager.cc:1428] T 711abfa18eee412a84905888b8994321 P bcf36ad493654e22a988b3d5fa661f10: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:36:49.845470 16570 tablet_service.cc:1467] Processing CreateTablet for tablet 711abfa18eee412a84905888b8994321 (DEFAULT_TABLE table=test-table [id=db26bf9b29ba4d6c989107fce86c6e6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:36:49.846426 16570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 711abfa18eee412a84905888b8994321. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:36:49.846182 16702 tablet_service.cc:1467] Processing CreateTablet for tablet 711abfa18eee412a84905888b8994321 (DEFAULT_TABLE table=test-table [id=db26bf9b29ba4d6c989107fce86c6e6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:36:49.847115 16702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 711abfa18eee412a84905888b8994321. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:36:49.852618 16781 tablet_bootstrap.cc:492] T 711abfa18eee412a84905888b8994321 P fb9f7e48bd2a42b0ad0cee0ee135f636: Bootstrap starting.
I20260927 02:36:49.852958 16782 tablet_bootstrap.cc:492] T 711abfa18eee412a84905888b8994321 P 060b505faf464a789ccc22f1bf18fa17: Bootstrap starting.
I20260927 02:36:49.854943 16781 tablet_bootstrap.cc:654] T 711abfa18eee412a84905888b8994321 P fb9f7e48bd2a42b0ad0cee0ee135f636: Neither blocks nor log segments found. Creating new log.
I20260927 02:36:49.855037 16782 tablet_bootstrap.cc:654] T 711abfa18eee412a84905888b8994321 P 060b505faf464a789ccc22f1bf18fa17: Neither blocks nor log segments found. Creating new log.
I20260927 02:36:49.855635 16781 log.cc:824] T 711abfa18eee412a84905888b8994321 P fb9f7e48bd2a42b0ad0cee0ee135f636: Log is configured to *not* fsync() on all Append() calls
I20260927 02:36:49.855995 16782 log.cc:824] T 711abfa18eee412a84905888b8994321 P 060b505faf464a789ccc22f1bf18fa17: Log is configured to *not* fsync() on all Append() calls
I20260927 02:36:49.857612 16781 tablet_bootstrap.cc:492] T 711abfa18eee412a84905888b8994321 P fb9f7e48bd2a42b0ad0cee0ee135f636: No bootstrap required, opened a new log
I20260927 02:36:49.857818 16781 ts_tablet_manager.cc:1397] T 711abfa18eee412a84905888b8994321 P fb9f7e48bd2a42b0ad0cee0ee135f636: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 02:36:49.857944 16782 tablet_bootstrap.cc:492] T 711abfa18eee412a84905888b8994321 P 060b505faf464a789ccc22f1bf18fa17: No bootstrap required, opened a new log
I20260927 02:36:49.858122 16782 ts_tablet_manager.cc:1397] T 711abfa18eee412a84905888b8994321 P 060b505faf464a789ccc22f1bf18fa17: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 02:36:49.860812 16781 raft_consensus.cc:348] T 711abfa18eee412a84905888b8994321 P fb9f7e48bd2a42b0ad0cee0ee135f636 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9f7e48bd2a42b0ad0cee0ee135f636" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } } peers { permanent_uuid: "060b505faf464a789ccc22f1bf18fa17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35341 } } peers { permanent_uuid: "bcf36ad493654e22a988b3d5fa661f10" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46463 } }
I20260927 02:36:49.860852 16782 raft_consensus.cc:348] T 711abfa18eee412a84905888b8994321 P 060b505faf464a789ccc22f1bf18fa17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9f7e48bd2a42b0ad0cee0ee135f636" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } } peers { permanent_uuid: "060b505faf464a789ccc22f1bf18fa17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35341 } } peers { permanent_uuid: "bcf36ad493654e22a988b3d5fa661f10" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46463 } }
I20260927 02:36:49.861102 16782 raft_consensus.cc:374] T 711abfa18eee412a84905888b8994321 P 060b505faf464a789ccc22f1bf18fa17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:36:49.861107 16781 raft_consensus.cc:374] T 711abfa18eee412a84905888b8994321 P fb9f7e48bd2a42b0ad0cee0ee135f636 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:36:49.861164 16781 raft_consensus.cc:729] T 711abfa18eee412a84905888b8994321 P fb9f7e48bd2a42b0ad0cee0ee135f636 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb9f7e48bd2a42b0ad0cee0ee135f636, State: Initialized, Role: FOLLOWER
I20260927 02:36:49.861164 16782 raft_consensus.cc:729] T 711abfa18eee412a84905888b8994321 P 060b505faf464a789ccc22f1bf18fa17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 060b505faf464a789ccc22f1bf18fa17, State: Initialized, Role: FOLLOWER
I20260927 02:36:49.861578 16782 consensus_queue.cc:260] T 711abfa18eee412a84905888b8994321 P 060b505faf464a789ccc22f1bf18fa17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9f7e48bd2a42b0ad0cee0ee135f636" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } } peers { permanent_uuid: "060b505faf464a789ccc22f1bf18fa17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35341 } } peers { permanent_uuid: "bcf36ad493654e22a988b3d5fa661f10" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46463 } }
I20260927 02:36:49.861590 16781 consensus_queue.cc:260] T 711abfa18eee412a84905888b8994321 P fb9f7e48bd2a42b0ad0cee0ee135f636 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9f7e48bd2a42b0ad0cee0ee135f636" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } } peers { permanent_uuid: "060b505faf464a789ccc22f1bf18fa17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35341 } } peers { permanent_uuid: "bcf36ad493654e22a988b3d5fa661f10" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46463 } }
I20260927 02:36:49.862517 16781 ts_tablet_manager.cc:1428] T 711abfa18eee412a84905888b8994321 P fb9f7e48bd2a42b0ad0cee0ee135f636: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 02:36:49.862528 16782 ts_tablet_manager.cc:1428] T 711abfa18eee412a84905888b8994321 P 060b505faf464a789ccc22f1bf18fa17: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260927 02:36:49.862696 16768 heartbeater.cc:503] Master 127.0.98.62:39525 was elected leader, sending a full tablet report...
I20260927 02:36:49.863619 16636 heartbeater.cc:503] Master 127.0.98.62:39525 was elected leader, sending a full tablet report...
W20260927 02:36:49.875865 16637 tablet.cc:2369] T 711abfa18eee412a84905888b8994321 P fb9f7e48bd2a42b0ad0cee0ee135f636: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:36:49.930214 16786 raft_consensus.cc:482] T 711abfa18eee412a84905888b8994321 P 060b505faf464a789ccc22f1bf18fa17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:36:49.930424 16786 raft_consensus.cc:504] T 711abfa18eee412a84905888b8994321 P 060b505faf464a789ccc22f1bf18fa17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9f7e48bd2a42b0ad0cee0ee135f636" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } } peers { permanent_uuid: "060b505faf464a789ccc22f1bf18fa17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35341 } } peers { permanent_uuid: "bcf36ad493654e22a988b3d5fa661f10" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46463 } }
I20260927 02:36:49.931507 16786 leader_election.cc:288] T 711abfa18eee412a84905888b8994321 P 060b505faf464a789ccc22f1bf18fa17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb9f7e48bd2a42b0ad0cee0ee135f636 (127.0.98.2:40749), bcf36ad493654e22a988b3d5fa661f10 (127.0.98.1:46463)
I20260927 02:36:49.935055 16770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:49.931780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40749 (local address 127.0.98.3:48461)
0927 02:36:49.931889 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:49.931903 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:49.932015 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:49.932288 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:36:49.932291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:36:49.932310 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:36:49.932443 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:49.932449 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:49.933211 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:49.933214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:49.933985 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:49.933993 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:49.934653 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:49.934657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:49.934758 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:49.934775 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 02:36:49.934838 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 02:36:49.934906 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"spinlock_wait_cycles":6272}
I20260927 02:36:49.935053 16777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:49.931865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48461 (local address 127.0.98.2:40749)
0927 02:36:49.931992 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:49.931997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:49.932014 (+ 17us) server_negotiation.cc:407] Connection header received
0927 02:36:49.932114 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:49.932117 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:49.932163 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:49.932254 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:36:49.932569 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:49.933108 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:49.934121 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:49.934540 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:49.934633 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:49.934696 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:49.934871 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 02:36:49.934939 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 02:36:49.935441 16506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:49.931936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58313 (local address 127.0.98.1:46463)
0927 02:36:49.932053 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:49.932058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:49.932359 (+ 301us) server_negotiation.cc:407] Connection header received
0927 02:36:49.932481 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:49.932483 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:49.932547 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:49.932628 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:36:49.932883 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:49.933443 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:49.934449 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:49.934880 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:49.934972 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:49.935134 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:49.935253 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 02:36:49.935329 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 02:36:49.935760 16787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:49.931780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46463 (local address 127.0.98.3:58313)
0927 02:36:49.932258 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:49.932274 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 02:36:49.932378 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:36:49.932650 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:36:49.932653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:36:49.932667 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:36:49.932778 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:49.932783 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:49.933550 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:49.933553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:49.934330 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:36:49.934339 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:36:49.935002 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:36:49.935005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:36:49.935095 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:49.935115 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 02:36:49.935578 (+ 463us) client_negotiation.cc:240] Negotiation successful
0927 02:36:49.935652 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"mutex_wait_us":110,"spinlock_wait_cycles":1280,"thread_start_us":130,"threads_started":1}
I20260927 02:36:49.935743 16590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "711abfa18eee412a84905888b8994321" candidate_uuid: "060b505faf464a789ccc22f1bf18fa17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb9f7e48bd2a42b0ad0cee0ee135f636" is_pre_election: true
I20260927 02:36:49.936050 16590 raft_consensus.cc:2454] T 711abfa18eee412a84905888b8994321 P fb9f7e48bd2a42b0ad0cee0ee135f636 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 060b505faf464a789ccc22f1bf18fa17 in term 0.
I20260927 02:36:49.936302 16452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "711abfa18eee412a84905888b8994321" candidate_uuid: "060b505faf464a789ccc22f1bf18fa17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf36ad493654e22a988b3d5fa661f10" is_pre_election: true
I20260927 02:36:49.936597 16452 raft_consensus.cc:2454] T 711abfa18eee412a84905888b8994321 P bcf36ad493654e22a988b3d5fa661f10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 060b505faf464a789ccc22f1bf18fa17 in term 0.
I20260927 02:36:49.936563 16657 leader_election.cc:302] T 711abfa18eee412a84905888b8994321 P 060b505faf464a789ccc22f1bf18fa17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 060b505faf464a789ccc22f1bf18fa17, fb9f7e48bd2a42b0ad0cee0ee135f636; no voters:
I20260927 02:36:49.936865 16786 raft_consensus.cc:2781] T 711abfa18eee412a84905888b8994321 P 060b505faf464a789ccc22f1bf18fa17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:36:49.936995 16786 raft_consensus.cc:482] T 711abfa18eee412a84905888b8994321 P 060b505faf464a789ccc22f1bf18fa17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:36:49.937054 16786 raft_consensus.cc:3037] T 711abfa18eee412a84905888b8994321 P 060b505faf464a789ccc22f1bf18fa17 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:36:49.938086 16786 raft_consensus.cc:504] T 711abfa18eee412a84905888b8994321 P 060b505faf464a789ccc22f1bf18fa17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9f7e48bd2a42b0ad0cee0ee135f636" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } } peers { permanent_uuid: "060b505faf464a789ccc22f1bf18fa17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35341 } } peers { permanent_uuid: "bcf36ad493654e22a988b3d5fa661f10" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46463 } }
I20260927 02:36:49.938409 16786 leader_election.cc:288] T 711abfa18eee412a84905888b8994321 P 060b505faf464a789ccc22f1bf18fa17 [CANDIDATE]: Term 1 election: Requested vote from peers fb9f7e48bd2a42b0ad0cee0ee135f636 (127.0.98.2:40749), bcf36ad493654e22a988b3d5fa661f10 (127.0.98.1:46463)
I20260927 02:36:49.938763 16590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "711abfa18eee412a84905888b8994321" candidate_uuid: "060b505faf464a789ccc22f1bf18fa17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb9f7e48bd2a42b0ad0cee0ee135f636"
I20260927 02:36:49.938753 16452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "711abfa18eee412a84905888b8994321" candidate_uuid: "060b505faf464a789ccc22f1bf18fa17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf36ad493654e22a988b3d5fa661f10"
I20260927 02:36:49.938884 16452 raft_consensus.cc:3037] T 711abfa18eee412a84905888b8994321 P bcf36ad493654e22a988b3d5fa661f10 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:36:49.938876 16590 raft_consensus.cc:3037] T 711abfa18eee412a84905888b8994321 P fb9f7e48bd2a42b0ad0cee0ee135f636 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:36:49.939874 16452 raft_consensus.cc:2454] T 711abfa18eee412a84905888b8994321 P bcf36ad493654e22a988b3d5fa661f10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 060b505faf464a789ccc22f1bf18fa17 in term 1.
I20260927 02:36:49.939908 16590 raft_consensus.cc:2454] T 711abfa18eee412a84905888b8994321 P fb9f7e48bd2a42b0ad0cee0ee135f636 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 060b505faf464a789ccc22f1bf18fa17 in term 1.
I20260927 02:36:49.940178 16658 leader_election.cc:302] T 711abfa18eee412a84905888b8994321 P 060b505faf464a789ccc22f1bf18fa17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 060b505faf464a789ccc22f1bf18fa17, bcf36ad493654e22a988b3d5fa661f10, fb9f7e48bd2a42b0ad0cee0ee135f636; no voters:
I20260927 02:36:49.940328 16786 raft_consensus.cc:2781] T 711abfa18eee412a84905888b8994321 P 060b505faf464a789ccc22f1bf18fa17 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:36:49.940531 16786 raft_consensus.cc:686] T 711abfa18eee412a84905888b8994321 P 060b505faf464a789ccc22f1bf18fa17 [term 1 LEADER]: Becoming Leader. State: Replica: 060b505faf464a789ccc22f1bf18fa17, State: Running, Role: LEADER
I20260927 02:36:49.940826 16786 consensus_queue.cc:237] T 711abfa18eee412a84905888b8994321 P 060b505faf464a789ccc22f1bf18fa17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9f7e48bd2a42b0ad0cee0ee135f636" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } } peers { permanent_uuid: "060b505faf464a789ccc22f1bf18fa17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35341 } } peers { permanent_uuid: "bcf36ad493654e22a988b3d5fa661f10" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46463 } }
W20260927 02:36:49.941653 16499 tablet.cc:2369] T 711abfa18eee412a84905888b8994321 P bcf36ad493654e22a988b3d5fa661f10: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:36:49.944029 16309 catalog_manager.cc:5495] T 711abfa18eee412a84905888b8994321 P 060b505faf464a789ccc22f1bf18fa17 reported cstate change: term changed from 0 to 1, leader changed from <none> to 060b505faf464a789ccc22f1bf18fa17 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "060b505faf464a789ccc22f1bf18fa17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9f7e48bd2a42b0ad0cee0ee135f636" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "060b505faf464a789ccc22f1bf18fa17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf36ad493654e22a988b3d5fa661f10" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46463 } health_report { overall_health: UNKNOWN } } }
I20260927 02:36:49.974326 16779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:49.970428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43818 (local address 127.0.98.3:35341)
0927 02:36:49.970558 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 02:36:49.970563 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:36:49.970579 (+ 16us) server_negotiation.cc:407] Connection header received
0927 02:36:49.970719 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:36:49.970721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:36:49.970787 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:36:49.970884 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:36:49.971101 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:49.971845 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:49.972817 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:36:49.973160 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:36:49.973256 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:36:49.973472 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:36:49.973958 (+ 486us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:36:49.974060 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 02:36:49.974110 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 02:36:49.974167 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 02:36:49.980046 16590 raft_consensus.cc:1261] T 711abfa18eee412a84905888b8994321 P fb9f7e48bd2a42b0ad0cee0ee135f636 [term 1 FOLLOWER]: Refusing update from remote peer 060b505faf464a789ccc22f1bf18fa17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:36:49.980047 16452 raft_consensus.cc:1261] T 711abfa18eee412a84905888b8994321 P bcf36ad493654e22a988b3d5fa661f10 [term 1 FOLLOWER]: Refusing update from remote peer 060b505faf464a789ccc22f1bf18fa17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:36:49.980656 16788 consensus_queue.cc:1035] T 711abfa18eee412a84905888b8994321 P 060b505faf464a789ccc22f1bf18fa17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf36ad493654e22a988b3d5fa661f10" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:36:49.980818 16786 consensus_queue.cc:1035] T 711abfa18eee412a84905888b8994321 P 060b505faf464a789ccc22f1bf18fa17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb9f7e48bd2a42b0ad0cee0ee135f636" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:36:49.987054 16792 mvcc.cc:205] Tried to move back new op lower bound from 7333792194471464960 to 7333792194320244736. Current Snapshot: MvccSnapshot[applied={T|T < 7333792194471464960}]
I20260927 02:36:49.987874 16793 mvcc.cc:205] Tried to move back new op lower bound from 7333792194471464960 to 7333792194320244736. Current Snapshot: MvccSnapshot[applied={T|T < 7333792194471464960}]
I20260927 02:36:49.987972 16791 mvcc.cc:205] Tried to move back new op lower bound from 7333792194471464960 to 7333792194320244736. Current Snapshot: MvccSnapshot[applied={T|T < 7333792194471464960}]
I20260927 02:36:49.996848 16310 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:36:49.997071 16310 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:36:49.999226 16310 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836: Sending DeleteTablet for 3 replicas of tablet 711abfa18eee412a84905888b8994321
I20260927 02:36:49.999806 16570 tablet_service.cc:1514] Processing DeleteTablet for tablet 711abfa18eee412a84905888b8994321 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:36:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38216
I20260927 02:36:49.999907 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 16348
I20260927 02:36:49.999902 16432 tablet_service.cc:1514] Processing DeleteTablet for tablet 711abfa18eee412a84905888b8994321 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:36:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48516
I20260927 02:36:50.002084 16702 tablet_service.cc:1514] Processing DeleteTablet for tablet 711abfa18eee412a84905888b8994321 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:36:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43804
I20260927 02:36:50.002523 16801 tablet_replica.cc:330] T 711abfa18eee412a84905888b8994321 P 060b505faf464a789ccc22f1bf18fa17: stopping tablet replica
I20260927 02:36:50.002825 16801 raft_consensus.cc:2229] T 711abfa18eee412a84905888b8994321 P 060b505faf464a789ccc22f1bf18fa17 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:36:50.003242 16801 raft_consensus.cc:2258] T 711abfa18eee412a84905888b8994321 P 060b505faf464a789ccc22f1bf18fa17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:36:50.004673 16801 ts_tablet_manager.cc:1905] T 711abfa18eee412a84905888b8994321 P 060b505faf464a789ccc22f1bf18fa17: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 02:36:50.004961 16297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46463 (error 108)
W20260927 02:36:50.005978 16778 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:36:50.005726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46463 (local address 127.0.0.1:48532)
0927 02:36:50.005841 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:50.005904 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46463: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 02:36:50.006201 16800 tablet_replica.cc:330] T 711abfa18eee412a84905888b8994321 P fb9f7e48bd2a42b0ad0cee0ee135f636: stopping tablet replica
I20260927 02:36:50.006467 16800 raft_consensus.cc:2229] T 711abfa18eee412a84905888b8994321 P fb9f7e48bd2a42b0ad0cee0ee135f636 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260927 02:36:50.006155 16297 catalog_manager.cc:4584] TS bcf36ad493654e22a988b3d5fa661f10 (127.0.98.1:46463): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 711abfa18eee412a84905888b8994321: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46463: connect: Connection reset by peer (error 104)
I20260927 02:36:50.006748 16800 raft_consensus.cc:2258] T 711abfa18eee412a84905888b8994321 P fb9f7e48bd2a42b0ad0cee0ee135f636 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:36:50.006901 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 16503
I20260927 02:36:50.007233 16801 ts_tablet_manager.cc:1918] T 711abfa18eee412a84905888b8994321 P 060b505faf464a789ccc22f1bf18fa17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:36:50.007349 16801 log.cc:1196] T 711abfa18eee412a84905888b8994321 P 060b505faf464a789ccc22f1bf18fa17: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-2/wal/wals/711abfa18eee412a84905888b8994321
I20260927 02:36:50.007635 16801 ts_tablet_manager.cc:1939] T 711abfa18eee412a84905888b8994321 P 060b505faf464a789ccc22f1bf18fa17: Deleting consensus metadata
I20260927 02:36:50.008623 16297 catalog_manager.cc:4841] TS 060b505faf464a789ccc22f1bf18fa17 (127.0.98.3:35341): tablet 711abfa18eee412a84905888b8994321 (table test-table [id=db26bf9b29ba4d6c989107fce86c6e6f]) successfully deleted
I20260927 02:36:50.012231 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 16639
I20260927 02:36:50.012416 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:36:50.012186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40749 (local address 127.0.0.1:38220)
0927 02:36:50.012300 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 02:36:50.012348 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:40749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 02:36:50.016877 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 16277
I20260927 02:37:10.022070 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39525
--webserver_interface=127.0.98.62
--webserver_port=35165
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39525
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:37:10.128327 16803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:37:10.128571 16803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:37:10.128608 16803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:37:10.132509 16803 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 02:37:10.132560 16803 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 02:37:10.132573 16803 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 02:37:10.132584 16803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:37:10.132601 16803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:37:10.132617 16803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:37:10.132635 16803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:37:10.132663 16803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:37:10.137960 16803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:37:10.139047 16803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:37:10.140067 16803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:37:10.148535 16803 server_base.cc:1034] running on GCE node
W20260927 02:37:10.148519 16808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:10.148700 16811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:10.148518 16809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:10.149303 16803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:37:10.150305 16803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:37:10.151453 16803 hybrid_clock.cc:648] HybridClock initialized: now 1790476630151434 us; error 29 us; skew 500 ppm
Sep 27 02:37:10 dist-test-slave-grlk krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476630, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:37:10.153900 16803 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:37:10.154731 16803 webserver.cc:460] Webserver started at http://127.0.98.62:35165/ using document root <none> and password file <none>
I20260927 02:37:10.155239 16803 fs_manager.cc:359] Metadata directory not provided
I20260927 02:37:10.155303 16803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:37:10.158870 16803 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:37:10.160781 16818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:10.161880 16803 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:37:10.161993 16803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "4f19bae86c5b42b0b6a725e5b1b20836"
format_stamp: "Formatted at 2026-09-27 02:36:27 on dist-test-slave-grlk"
server_key: "b0699c1d4624bea5a4f5c11aa99316e2"
server_key_iv: "2fa3876119f1a2b575804cc2a56b3967"
server_key_version: "encryptionkey@0"
I20260927 02:37:10.162526 16803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:37:10.180169 16803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:37:10.182768 16803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:37:10.182940 16803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:37:10.189380 16803 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39525
I20260927 02:37:10.189612 16870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39525 every 8 connection(s)
I20260927 02:37:10.190341 16803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:37:10.194196 16871 sys_catalog.cc:263] Verifying existing consensus state
I20260927 02:37:10.195670 16871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836: Bootstrap starting.
I20260927 02:37:10.197451 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 16803
I20260927 02:37:10.197893 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46463
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39525
--webserver_port=45407
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:37:10.212466 16871 log.cc:824] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836: Log is configured to *not* fsync() on all Append() calls
I20260927 02:37:10.223055 16871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:37:10.223356 16871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836: Bootstrap complete.
I20260927 02:37:10.226967 16871 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f19bae86c5b42b0b6a725e5b1b20836" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39525 } }
I20260927 02:37:10.227319 16871 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f19bae86c5b42b0b6a725e5b1b20836, State: Initialized, Role: FOLLOWER
I20260927 02:37:10.227777 16871 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f19bae86c5b42b0b6a725e5b1b20836" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39525 } }
I20260927 02:37:10.227888 16871 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:37:10.227949 16871 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:37:10.228032 16871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [term 2 FOLLOWER]: Advancing to term 3
I20260927 02:37:10.229768 16871 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f19bae86c5b42b0b6a725e5b1b20836" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39525 } }
I20260927 02:37:10.230053 16871 leader_election.cc:302] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f19bae86c5b42b0b6a725e5b1b20836; no voters:
I20260927 02:37:10.230311 16871 leader_election.cc:288] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 02:37:10.230396 16875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 02:37:10.230578 16875 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [term 3 LEADER]: Becoming Leader. State: Replica: 4f19bae86c5b42b0b6a725e5b1b20836, State: Running, Role: LEADER
I20260927 02:37:10.230926 16875 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f19bae86c5b42b0b6a725e5b1b20836" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39525 } }
I20260927 02:37:10.231433 16871 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:37:10.232275 16877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f19bae86c5b42b0b6a725e5b1b20836. Latest consensus state: current_term: 3 leader_uuid: "4f19bae86c5b42b0b6a725e5b1b20836" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f19bae86c5b42b0b6a725e5b1b20836" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39525 } } }
I20260927 02:37:10.232313 16876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "4f19bae86c5b42b0b6a725e5b1b20836" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f19bae86c5b42b0b6a725e5b1b20836" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39525 } } }
I20260927 02:37:10.232393 16877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [sys.catalog]: This master's current role is: LEADER
I20260927 02:37:10.232416 16876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836 [sys.catalog]: This master's current role is: LEADER
I20260927 02:37:10.235991 16890 catalog_manager.cc:1259] Loaded cluster ID: 6620759ac746484aa0f68da98198dbed
I20260927 02:37:10.236106 16890 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836: loading cluster ID for follower catalog manager: success
I20260927 02:37:10.237882 16890 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4f19bae86c5b42b0b6a725e5b1b20836: acquiring CA information for follower catalog manager: success
F20260927 02:37:10.238775 16890 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f8bacf379ad google::LogMessage::Fail() at ??:0
@ 0x7f8bacf3a88a google::LogMessage::SendToLog() at ??:0
@ 0x7f8bacf374de google::LogMessage::Flush() at ??:0
@ 0x7f8bacf38a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f8bafbfb18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f8bb789c989 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7f8bb7879023 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7f8bb787954b kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7f8bb7872652 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7f8bb78719e9 _ZZN4kudu6master21CatalogManagerBgTasks4InitEvENKUlvE_clEv at ??:0
@ 0x7f8bb78b8541 _ZNSt17_Function_handlerIFvvEZN4kudu6master21CatalogManagerBgTasks4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f8baf17a512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f8bb07be6db start_thread at ??:0
@ 0x7f8baba3561f clone at ??:0
W20260927 02:37:10.316949 16873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:37:10.317209 16873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:37:10.317269 16873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:37:10.321146 16873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:37:10.321194 16873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:37:10.321269 16873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:37:10.326417 16873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46463
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=45407
--tserver_master_addrs=127.0.98.62:39525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:37:10.327411 16873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:37:10.328342 16873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:37:10.337394 16914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:10.337380 16911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:10.337383 16912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:10.337632 16873 server_base.cc:1034] running on GCE node
I20260927 02:37:10.338217 16873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:37:10.338930 16873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:37:10.340118 16873 hybrid_clock.cc:648] HybridClock initialized: now 1790476630340093 us; error 39 us; skew 500 ppm
Sep 27 02:37:10 dist-test-slave-grlk krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476630, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:37:10.343226 16873 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:37:10.344046 16873 webserver.cc:460] Webserver started at http://127.0.98.1:45407/ using document root <none> and password file <none>
I20260927 02:37:10.344547 16873 fs_manager.cc:359] Metadata directory not provided
I20260927 02:37:10.344604 16873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:37:10.348263 16873 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:37:10.350356 16921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:10.351536 16873 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:37:10.351642 16873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-0/wal
uuid: "bcf36ad493654e22a988b3d5fa661f10"
format_stamp: "Formatted at 2026-09-27 02:36:28 on dist-test-slave-grlk"
server_key: "bd1dae9cfba5669d741e56ce9325ef77"
server_key_iv: "845aaf2fe601841a23a95204480fecfd"
server_key_version: "encryptionkey@0"
I20260927 02:37:10.352130 16873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:37:10.364365 16873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:37:10.367293 16873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:37:10.367455 16873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:37:10.367980 16873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:37:10.369021 16928 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 02:37:10.374140 16873 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 02:37:10.374214 16873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260927 02:37:10.374284 16873 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 02:37:10.375976 16873 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 02:37:10.376025 16873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 02:37:10.376315 16928 tablet_bootstrap.cc:492] T 711abfa18eee412a84905888b8994321 P bcf36ad493654e22a988b3d5fa661f10: Bootstrap starting.
I20260927 02:37:10.386286 16873 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46463
I20260927 02:37:10.386477 17035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46463 every 8 connection(s)
I20260927 02:37:10.387259 16873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:37:10.393929 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 16873
I20260927 02:37:10.394330 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40749
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39525
--webserver_port=33353
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:37:10.398545 16928 log.cc:824] T 711abfa18eee412a84905888b8994321 P bcf36ad493654e22a988b3d5fa661f10: Log is configured to *not* fsync() on all Append() calls
I20260927 02:37:10.404883 16928 tablet_bootstrap.cc:492] T 711abfa18eee412a84905888b8994321 P bcf36ad493654e22a988b3d5fa661f10: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:37:10.405205 16928 tablet_bootstrap.cc:492] T 711abfa18eee412a84905888b8994321 P bcf36ad493654e22a988b3d5fa661f10: Bootstrap complete.
I20260927 02:37:10.406081 16928 ts_tablet_manager.cc:1397] T 711abfa18eee412a84905888b8994321 P bcf36ad493654e22a988b3d5fa661f10: Time spent bootstrapping tablet: real 0.030s user 0.023s sys 0.004s
I20260927 02:37:10.408272 16928 raft_consensus.cc:348] T 711abfa18eee412a84905888b8994321 P bcf36ad493654e22a988b3d5fa661f10 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9f7e48bd2a42b0ad0cee0ee135f636" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } } peers { permanent_uuid: "060b505faf464a789ccc22f1bf18fa17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35341 } } peers { permanent_uuid: "bcf36ad493654e22a988b3d5fa661f10" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46463 } }
I20260927 02:37:10.408706 16928 raft_consensus.cc:729] T 711abfa18eee412a84905888b8994321 P bcf36ad493654e22a988b3d5fa661f10 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf36ad493654e22a988b3d5fa661f10, State: Initialized, Role: FOLLOWER
I20260927 02:37:10.409168 16928 consensus_queue.cc:260] T 711abfa18eee412a84905888b8994321 P bcf36ad493654e22a988b3d5fa661f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9f7e48bd2a42b0ad0cee0ee135f636" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } } peers { permanent_uuid: "060b505faf464a789ccc22f1bf18fa17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35341 } } peers { permanent_uuid: "bcf36ad493654e22a988b3d5fa661f10" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46463 } }
I20260927 02:37:10.409940 16928 ts_tablet_manager.cc:1428] T 711abfa18eee412a84905888b8994321 P bcf36ad493654e22a988b3d5fa661f10: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20260927 02:37:10.451304 17038 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:37:10.389125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.1:58257)
0927 02:37:10.389583 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:10.389635 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:10.390413 (+ 778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:10.450402 (+ 59989us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":124,"threads_started":1}
W20260927 02:37:10.451853 17036 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39525 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:39525: Client connection negotiation failed: client connection to 127.0.98.62:39525: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 02:37:10.452526 17038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:10.452280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.1:59299)
0927 02:37:10.452399 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:10.452466 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 02:37:10.453176 17038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:10.452977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.1:39247)
0927 02:37:10.453072 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:10.453120 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 02:37:10.453322 17036 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 02:37:10.510687 17039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:37:10.510908 17039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:37:10.510947 17039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:37:10.514807 17039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:37:10.514851 17039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:37:10.514905 17039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:37:10.519723 17039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40749
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33353
--tserver_master_addrs=127.0.98.62:39525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:37:10.520643 17039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:37:10.521580 17039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:37:10.530401 17046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:10.530496 17039 server_base.cc:1034] running on GCE node
W20260927 02:37:10.530424 17049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:10.530437 17047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:10.530990 17039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:37:10.531574 17039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:37:10.532733 17039 hybrid_clock.cc:648] HybridClock initialized: now 1790476630532720 us; error 28 us; skew 500 ppm
Sep 27 02:37:10 dist-test-slave-grlk krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476630, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:37:10.535269 17039 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:37:10.536038 17039 webserver.cc:460] Webserver started at http://127.0.98.2:33353/ using document root <none> and password file <none>
I20260927 02:37:10.536533 17039 fs_manager.cc:359] Metadata directory not provided
I20260927 02:37:10.536592 17039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:37:10.540082 17039 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:37:10.542029 17056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:10.543162 17039 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:37:10.543275 17039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-1/wal
uuid: "fb9f7e48bd2a42b0ad0cee0ee135f636"
format_stamp: "Formatted at 2026-09-27 02:36:28 on dist-test-slave-grlk"
server_key: "b572203f42735d94f070734bd05f9f55"
server_key_iv: "4767fea16b5ad0c79da83fc0524a3d1f"
server_key_version: "encryptionkey@0"
I20260927 02:37:10.543790 17039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:37:10.558281 17039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:37:10.561333 17039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:37:10.561504 17039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:37:10.562115 17039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:37:10.563210 17063 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 02:37:10.567118 17039 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 02:37:10.567181 17039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260927 02:37:10.567250 17039 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 02:37:10.568774 17039 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 02:37:10.568816 17039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 02:37:10.569068 17063 tablet_bootstrap.cc:492] T 711abfa18eee412a84905888b8994321 P fb9f7e48bd2a42b0ad0cee0ee135f636: Bootstrap starting.
I20260927 02:37:10.578018 17039 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40749
I20260927 02:37:10.578091 17170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40749 every 8 connection(s)
I20260927 02:37:10.578910 17039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:37:10.580967 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 17039
I20260927 02:37:10.581447 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35341
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39525
--webserver_port=36991
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:37:10.582221 17173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:10.580867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.2:48641)
0927 02:37:10.581364 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:10.581485 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":135,"threads_started":1}
W20260927 02:37:10.583156 17171 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39525 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:39525: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
I20260927 02:37:10.583819 17173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:10.583573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.2:57079)
0927 02:37:10.583701 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:10.583755 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 02:37:10.584393 17173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:10.584172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.2:51729)
0927 02:37:10.584280 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:10.584331 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 02:37:10.584547 17171 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 02:37:10.586949 17063 log.cc:824] T 711abfa18eee412a84905888b8994321 P fb9f7e48bd2a42b0ad0cee0ee135f636: Log is configured to *not* fsync() on all Append() calls
I20260927 02:37:10.595505 17063 tablet_bootstrap.cc:492] T 711abfa18eee412a84905888b8994321 P fb9f7e48bd2a42b0ad0cee0ee135f636: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:37:10.595902 17063 tablet_bootstrap.cc:492] T 711abfa18eee412a84905888b8994321 P fb9f7e48bd2a42b0ad0cee0ee135f636: Bootstrap complete.
I20260927 02:37:10.597306 17063 ts_tablet_manager.cc:1397] T 711abfa18eee412a84905888b8994321 P fb9f7e48bd2a42b0ad0cee0ee135f636: Time spent bootstrapping tablet: real 0.028s user 0.024s sys 0.004s
I20260927 02:37:10.600864 17063 raft_consensus.cc:348] T 711abfa18eee412a84905888b8994321 P fb9f7e48bd2a42b0ad0cee0ee135f636 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9f7e48bd2a42b0ad0cee0ee135f636" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } } peers { permanent_uuid: "060b505faf464a789ccc22f1bf18fa17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35341 } } peers { permanent_uuid: "bcf36ad493654e22a988b3d5fa661f10" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46463 } }
I20260927 02:37:10.601423 17063 raft_consensus.cc:729] T 711abfa18eee412a84905888b8994321 P fb9f7e48bd2a42b0ad0cee0ee135f636 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb9f7e48bd2a42b0ad0cee0ee135f636, State: Initialized, Role: FOLLOWER
I20260927 02:37:10.601910 17063 consensus_queue.cc:260] T 711abfa18eee412a84905888b8994321 P fb9f7e48bd2a42b0ad0cee0ee135f636 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9f7e48bd2a42b0ad0cee0ee135f636" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } } peers { permanent_uuid: "060b505faf464a789ccc22f1bf18fa17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35341 } } peers { permanent_uuid: "bcf36ad493654e22a988b3d5fa661f10" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46463 } }
I20260927 02:37:10.602833 17063 ts_tablet_manager.cc:1428] T 711abfa18eee412a84905888b8994321 P fb9f7e48bd2a42b0ad0cee0ee135f636: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 02:37:10.602993 17173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:10.602740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.2:53723)
0927 02:37:10.602856 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:10.602913 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 02:37:10.603593 17173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:10.603422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.2:35009)
0927 02:37:10.603507 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:10.603545 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 02:37:10.693552 17174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:37:10.693780 17174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:37:10.693820 17174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:37:10.697695 17174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:37:10.697741 17174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:37:10.697806 17174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:37:10.702747 17174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35341
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36991
--tserver_master_addrs=127.0.98.62:39525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:37:10.703614 17174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:37:10.704499 17174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:37:10.713244 17181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:10.713287 17182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:10.713287 17184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:10.713419 17174 server_base.cc:1034] running on GCE node
I20260927 02:37:10.713884 17174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:37:10.714407 17174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:37:10.715584 17174 hybrid_clock.cc:648] HybridClock initialized: now 1790476630715545 us; error 26 us; skew 500 ppm
Sep 27 02:37:10 dist-test-slave-grlk krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476630, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:37:10.717991 17174 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:37:10.718747 17174 webserver.cc:460] Webserver started at http://127.0.98.3:36991/ using document root <none> and password file <none>
I20260927 02:37:10.719228 17174 fs_manager.cc:359] Metadata directory not provided
I20260927 02:37:10.719290 17174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:37:10.722699 17174 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:37:10.724483 17191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:10.725603 17174 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:37:10.725708 17174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-2/wal
uuid: "060b505faf464a789ccc22f1bf18fa17"
format_stamp: "Formatted at 2026-09-27 02:36:28 on dist-test-slave-grlk"
server_key: "b1f068eccd152be1f1018a26a8a75225"
server_key_iv: "299479cda5846f2ea95e8cbdacd13760"
server_key_version: "encryptionkey@0"
I20260927 02:37:10.726182 17174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:37:10.743388 17174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:37:10.745901 17174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:37:10.746053 17174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:37:10.746557 17174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:37:10.747321 17174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:37:10.747385 17174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:10.747433 17174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:37:10.747463 17174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:10.755262 17174 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35341
I20260927 02:37:10.755280 17304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35341 every 8 connection(s)
I20260927 02:37:10.756134 17174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:37:10.758874 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 17174
I20260927 02:37:10.759634 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:10.758028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.3:60697)
0927 02:37:10.758443 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:10.758539 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":103,"threads_started":1}
W20260927 02:37:10.760038 17305 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39525 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:39525: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
I20260927 02:37:10.760552 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:10.760361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.3:54981)
0927 02:37:10.760462 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:10.760502 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 02:37:10.761077 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:10.760882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.3:43361)
0927 02:37:10.760989 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:10.761031 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 02:37:10.761217 17305 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 02:37:11.454191 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:11.453822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.1:44195)
0927 02:37:11.454038 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:11.454088 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":68,"threads_started":1}
I20260927 02:37:11.605094 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:11.604357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.2:38421)
0927 02:37:11.604780 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:11.604878 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":82,"threads_started":1}
I20260927 02:37:11.762028 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:11.761684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.3:53775)
0927 02:37:11.761894 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:11.761943 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":61,"threads_started":1}
I20260927 02:37:11.884716 17312 raft_consensus.cc:482] T 711abfa18eee412a84905888b8994321 P fb9f7e48bd2a42b0ad0cee0ee135f636 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:37:11.884963 17312 raft_consensus.cc:504] T 711abfa18eee412a84905888b8994321 P fb9f7e48bd2a42b0ad0cee0ee135f636 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9f7e48bd2a42b0ad0cee0ee135f636" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } } peers { permanent_uuid: "060b505faf464a789ccc22f1bf18fa17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35341 } } peers { permanent_uuid: "bcf36ad493654e22a988b3d5fa661f10" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46463 } }
I20260927 02:37:11.885985 17312 leader_election.cc:288] T 711abfa18eee412a84905888b8994321 P fb9f7e48bd2a42b0ad0cee0ee135f636 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 060b505faf464a789ccc22f1bf18fa17 (127.0.98.3:35341), bcf36ad493654e22a988b3d5fa661f10 (127.0.98.1:46463)
Sep 27 02:37:11 dist-test-slave-grlk krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476630, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
Sep 27 02:37:11 dist-test-slave-grlk krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476630, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260927 02:37:11.898589 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:11.886447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46463 (local address 127.0.98.2:45061)
0927 02:37:11.886712 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:11.886726 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:11.886829 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:11.887615 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:11.887625 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:11.888062 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:37:11.888589 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:11.888605 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:11.889860 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:11.889863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:11.890700 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:11.890709 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:11.891472 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:11.891476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:11.891597 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:11.892256 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:37:11.892282 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:37:11.893761 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:37:11.895776 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:37:11.895782 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:37:11.895815 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:37:11.896266 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:37:11.896922 (+ 656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:37:11.896925 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:37:11.896949 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:37:11.897032 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:37:11.897629 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:37:11.897632 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:37:11.897848 (+ 216us) client_negotiation.cc:769] Sending connection context
0927 02:37:11.898065 (+ 217us) client_negotiation.cc:240] Negotiation successful
0927 02:37:11.898389 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":128,"spinlock_wait_cycles":15232,"thread_start_us":56,"threads_started":1}
I20260927 02:37:11.898609 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:11.886625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45061 (local address 127.0.98.1:46463)
0927 02:37:11.887003 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:11.887009 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:11.887033 (+ 24us) server_negotiation.cc:407] Connection header received
0927 02:37:11.887163 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:11.887168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:11.887357 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:11.887507 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:11.888751 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:11.889734 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:11.890810 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:11.891264 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:11.891416 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:11.893947 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:11.893954 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:11.893972 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:37:11.894004 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:11.895654 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:11.896703 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:11.896708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:11.896714 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:11.896832 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:11.897119 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:11.897122 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:11.897124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:11.897540 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:11.897650 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:11.898180 (+ 530us) server_negotiation.cc:299] Negotiation successful
0927 02:37:11.898437 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":89,"threads_started":1}
I20260927 02:37:11.898619 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:11.886293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49293 (local address 127.0.98.3:35341)
0927 02:37:11.886805 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:11.886826 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:11.886873 (+ 47us) server_negotiation.cc:407] Connection header received
0927 02:37:11.887620 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:11.887625 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:11.887903 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:11.888135 (+ 232us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:11.888740 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:11.889726 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:11.890825 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:11.891288 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:11.891426 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:11.894297 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:11.894301 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:11.894322 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:37:11.894349 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:11.895751 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:11.896386 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:11.896412 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:11.896417 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:11.896564 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:11.896956 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:11.896959 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:11.896961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:11.897454 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:11.897597 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:11.898181 (+ 584us) server_negotiation.cc:299] Negotiation successful
0927 02:37:11.898437 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":80,"threads_started":1}
I20260927 02:37:11.898588 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:11.886166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35341 (local address 127.0.98.2:49293)
0927 02:37:11.886287 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:11.886321 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:11.886795 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:11.888102 (+ 1307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:11.888105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:11.888296 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:37:11.888590 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:11.888605 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:11.889854 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:11.889858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:11.890706 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:11.890715 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:11.891399 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:11.891402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:11.891576 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:11.892283 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:37:11.892295 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:37:11.894113 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:37:11.895854 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:37:11.895858 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:37:11.895860 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:37:11.896256 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:37:11.896656 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:37:11.896658 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:37:11.896660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:37:11.896835 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:37:11.897558 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:37:11.897563 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:37:11.897848 (+ 285us) client_negotiation.cc:769] Sending connection context
0927 02:37:11.898065 (+ 217us) client_negotiation.cc:240] Negotiation successful
0927 02:37:11.898453 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":170,"spinlock_wait_cycles":10240}
I20260927 02:37:11.900064 16990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "711abfa18eee412a84905888b8994321" candidate_uuid: "fb9f7e48bd2a42b0ad0cee0ee135f636" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "bcf36ad493654e22a988b3d5fa661f10" is_pre_election: true
I20260927 02:37:11.900353 16990 raft_consensus.cc:2454] T 711abfa18eee412a84905888b8994321 P bcf36ad493654e22a988b3d5fa661f10 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb9f7e48bd2a42b0ad0cee0ee135f636 in term 1.
W20260927 02:37:11.901105 17060 leader_election.cc:341] T 711abfa18eee412a84905888b8994321 P fb9f7e48bd2a42b0ad0cee0ee135f636 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 060b505faf464a789ccc22f1bf18fa17 (127.0.98.3:35341): Not found: Tablet not found: 711abfa18eee412a84905888b8994321
I20260927 02:37:11.901328 17060 leader_election.cc:302] T 711abfa18eee412a84905888b8994321 P fb9f7e48bd2a42b0ad0cee0ee135f636 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bcf36ad493654e22a988b3d5fa661f10, fb9f7e48bd2a42b0ad0cee0ee135f636; no voters: 060b505faf464a789ccc22f1bf18fa17
I20260927 02:37:11.901544 17312 raft_consensus.cc:2781] T 711abfa18eee412a84905888b8994321 P fb9f7e48bd2a42b0ad0cee0ee135f636 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 02:37:11.901635 17312 raft_consensus.cc:482] T 711abfa18eee412a84905888b8994321 P fb9f7e48bd2a42b0ad0cee0ee135f636 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:37:11.901692 17312 raft_consensus.cc:3037] T 711abfa18eee412a84905888b8994321 P fb9f7e48bd2a42b0ad0cee0ee135f636 [term 1 FOLLOWER]: Advancing to term 2
I20260927 02:37:11.903613 17312 raft_consensus.cc:504] T 711abfa18eee412a84905888b8994321 P fb9f7e48bd2a42b0ad0cee0ee135f636 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9f7e48bd2a42b0ad0cee0ee135f636" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } } peers { permanent_uuid: "060b505faf464a789ccc22f1bf18fa17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35341 } } peers { permanent_uuid: "bcf36ad493654e22a988b3d5fa661f10" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46463 } }
I20260927 02:37:11.903927 17312 leader_election.cc:288] T 711abfa18eee412a84905888b8994321 P fb9f7e48bd2a42b0ad0cee0ee135f636 [CANDIDATE]: Term 2 election: Requested vote from peers 060b505faf464a789ccc22f1bf18fa17 (127.0.98.3:35341), bcf36ad493654e22a988b3d5fa661f10 (127.0.98.1:46463)
I20260927 02:37:11.904299 16990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "711abfa18eee412a84905888b8994321" candidate_uuid: "fb9f7e48bd2a42b0ad0cee0ee135f636" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "bcf36ad493654e22a988b3d5fa661f10"
I20260927 02:37:11.904431 16990 raft_consensus.cc:3037] T 711abfa18eee412a84905888b8994321 P bcf36ad493654e22a988b3d5fa661f10 [term 1 FOLLOWER]: Advancing to term 2
W20260927 02:37:11.904522 17060 leader_election.cc:341] T 711abfa18eee412a84905888b8994321 P fb9f7e48bd2a42b0ad0cee0ee135f636 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 060b505faf464a789ccc22f1bf18fa17 (127.0.98.3:35341): Not found: Tablet not found: 711abfa18eee412a84905888b8994321
I20260927 02:37:11.906291 16990 raft_consensus.cc:2454] T 711abfa18eee412a84905888b8994321 P bcf36ad493654e22a988b3d5fa661f10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb9f7e48bd2a42b0ad0cee0ee135f636 in term 2.
I20260927 02:37:11.906539 17060 leader_election.cc:302] T 711abfa18eee412a84905888b8994321 P fb9f7e48bd2a42b0ad0cee0ee135f636 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bcf36ad493654e22a988b3d5fa661f10, fb9f7e48bd2a42b0ad0cee0ee135f636; no voters: 060b505faf464a789ccc22f1bf18fa17
I20260927 02:37:11.906675 17312 raft_consensus.cc:2781] T 711abfa18eee412a84905888b8994321 P fb9f7e48bd2a42b0ad0cee0ee135f636 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 02:37:11.906858 17312 raft_consensus.cc:686] T 711abfa18eee412a84905888b8994321 P fb9f7e48bd2a42b0ad0cee0ee135f636 [term 2 LEADER]: Becoming Leader. State: Replica: fb9f7e48bd2a42b0ad0cee0ee135f636, State: Running, Role: LEADER
I20260927 02:37:11.907150 17312 consensus_queue.cc:237] T 711abfa18eee412a84905888b8994321 P fb9f7e48bd2a42b0ad0cee0ee135f636 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9f7e48bd2a42b0ad0cee0ee135f636" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } } peers { permanent_uuid: "060b505faf464a789ccc22f1bf18fa17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35341 } } peers { permanent_uuid: "bcf36ad493654e22a988b3d5fa661f10" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46463 } }
I20260927 02:37:11.907455 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:11.907232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.2:55075)
0927 02:37:11.907345 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:11.907395 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 02:37:12.391403 16990 raft_consensus.cc:1261] T 711abfa18eee412a84905888b8994321 P bcf36ad493654e22a988b3d5fa661f10 [term 2 FOLLOWER]: Refusing update from remote peer fb9f7e48bd2a42b0ad0cee0ee135f636: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260927 02:37:12.392035 17312 consensus_queue.cc:1035] T 711abfa18eee412a84905888b8994321 P fb9f7e48bd2a42b0ad0cee0ee135f636 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf36ad493654e22a988b3d5fa661f10" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 02:37:12.394083 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:12.393708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.1:39639)
0927 02:37:12.393924 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:12.393987 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":67,"threads_started":1}
I20260927 02:37:12.396446 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:12.396190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.1:60885)
0927 02:37:12.396332 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:12.396385 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 02:37:12.396461 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:12.395836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.2:34873)
0927 02:37:12.396356 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:12.396402 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448}
W20260927 02:37:12.396813 17060 consensus_peers.cc:487] T 711abfa18eee412a84905888b8994321 P fb9f7e48bd2a42b0ad0cee0ee135f636 -> Peer 060b505faf464a789ccc22f1bf18fa17 (127.0.98.3:35341): Couldn't send request to peer 060b505faf464a789ccc22f1bf18fa17. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 711abfa18eee412a84905888b8994321. This is attempt 1: this message will repeat every 5th retry.
I20260927 02:37:12.762944 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:12.762598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.3:45927)
0927 02:37:12.762812 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:12.762859 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":65,"threads_started":1}
I20260927 02:37:12.860756 17325 ts_tablet_manager.cc:927] T 711abfa18eee412a84905888b8994321 P 060b505faf464a789ccc22f1bf18fa17: Initiating tablet copy from peer fb9f7e48bd2a42b0ad0cee0ee135f636 (127.0.98.2:40749)
I20260927 02:37:12.861368 17325 tablet_copy_client.cc:316] T 711abfa18eee412a84905888b8994321 P 060b505faf464a789ccc22f1bf18fa17: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.2:40749
Sep 27 02:37:12 dist-test-slave-grlk krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476630, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260927 02:37:12.870057 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:12.861840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40749 (local address 127.0.98.3:51519)
0927 02:37:12.861959 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:12.861980 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:12.862124 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:12.862549 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:12.862553 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:12.862910 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:37:12.863021 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:12.863027 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:12.863899 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:12.863902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:12.864506 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:12.864513 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:12.865073 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:12.865076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:12.865168 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:12.865671 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:37:12.865682 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:37:12.867018 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:37:12.868672 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:37:12.868679 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:37:12.868684 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:37:12.868896 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:37:12.869168 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:37:12.869170 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:37:12.869171 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:37:12.869242 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:37:12.869600 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:37:12.869605 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:37:12.869702 (+ 97us) client_negotiation.cc:769] Sending connection context
0927 02:37:12.869820 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 02:37:12.869940 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 02:37:12.870152 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:12.861966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51519 (local address 127.0.98.2:40749)
0927 02:37:12.862192 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:12.862198 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:12.862217 (+ 19us) server_negotiation.cc:407] Connection header received
0927 02:37:12.862266 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:12.862271 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:12.862432 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:12.862537 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:12.863110 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:12.863818 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:12.864587 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:12.864923 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:12.865036 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:12.867168 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:12.867177 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:12.867186 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:37:12.867212 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:12.868576 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:12.869007 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:12.869012 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:12.869013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:12.869091 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:12.869304 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:12.869306 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:12.869307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:12.869523 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:12.869618 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:12.869887 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 02:37:12.870000 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":67,"threads_started":1}
I20260927 02:37:12.871006 17145 tablet_copy_service.cc:140] P fb9f7e48bd2a42b0ad0cee0ee135f636: Received BeginTabletCopySession request for tablet 711abfa18eee412a84905888b8994321 from peer 060b505faf464a789ccc22f1bf18fa17 ({username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51519)
I20260927 02:37:12.871160 17145 tablet_copy_service.cc:161] P fb9f7e48bd2a42b0ad0cee0ee135f636: Beginning new tablet copy session on tablet 711abfa18eee412a84905888b8994321 from peer 060b505faf464a789ccc22f1bf18fa17 at {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51519: session id = 060b505faf464a789ccc22f1bf18fa17-711abfa18eee412a84905888b8994321
I20260927 02:37:12.872222 17145 tablet_copy_source_session.cc:205] T 711abfa18eee412a84905888b8994321 P fb9f7e48bd2a42b0ad0cee0ee135f636: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 02:37:12.875763 17325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 711abfa18eee412a84905888b8994321. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:37:12.881839 17325 tablet_copy_client.cc:791] T 711abfa18eee412a84905888b8994321 P 060b505faf464a789ccc22f1bf18fa17: tablet copy: Starting download of 0 data blocks...
I20260927 02:37:12.882009 17325 tablet_copy_client.cc:655] T 711abfa18eee412a84905888b8994321 P 060b505faf464a789ccc22f1bf18fa17: tablet copy: Starting download of 1 WAL segments...
I20260927 02:37:12.883481 17325 tablet_copy_client.cc:531] T 711abfa18eee412a84905888b8994321 P 060b505faf464a789ccc22f1bf18fa17: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 02:37:12.885011 17325 tablet_bootstrap.cc:492] T 711abfa18eee412a84905888b8994321 P 060b505faf464a789ccc22f1bf18fa17: Bootstrap starting.
I20260927 02:37:12.889979 17325 log.cc:824] T 711abfa18eee412a84905888b8994321 P 060b505faf464a789ccc22f1bf18fa17: Log is configured to *not* fsync() on all Append() calls
I20260927 02:37:12.896219 17325 tablet_bootstrap.cc:492] T 711abfa18eee412a84905888b8994321 P 060b505faf464a789ccc22f1bf18fa17: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:37:12.896476 17325 tablet_bootstrap.cc:492] T 711abfa18eee412a84905888b8994321 P 060b505faf464a789ccc22f1bf18fa17: Bootstrap complete.
I20260927 02:37:12.896787 17325 ts_tablet_manager.cc:1397] T 711abfa18eee412a84905888b8994321 P 060b505faf464a789ccc22f1bf18fa17: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260927 02:37:12.899083 17325 raft_consensus.cc:348] T 711abfa18eee412a84905888b8994321 P 060b505faf464a789ccc22f1bf18fa17 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9f7e48bd2a42b0ad0cee0ee135f636" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } } peers { permanent_uuid: "060b505faf464a789ccc22f1bf18fa17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35341 } } peers { permanent_uuid: "bcf36ad493654e22a988b3d5fa661f10" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46463 } }
I20260927 02:37:12.899392 17325 raft_consensus.cc:729] T 711abfa18eee412a84905888b8994321 P 060b505faf464a789ccc22f1bf18fa17 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 060b505faf464a789ccc22f1bf18fa17, State: Initialized, Role: FOLLOWER
I20260927 02:37:12.899835 17325 consensus_queue.cc:260] T 711abfa18eee412a84905888b8994321 P 060b505faf464a789ccc22f1bf18fa17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9f7e48bd2a42b0ad0cee0ee135f636" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } } peers { permanent_uuid: "060b505faf464a789ccc22f1bf18fa17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35341 } } peers { permanent_uuid: "bcf36ad493654e22a988b3d5fa661f10" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46463 } }
I20260927 02:37:12.900647 17325 ts_tablet_manager.cc:1428] T 711abfa18eee412a84905888b8994321 P 060b505faf464a789ccc22f1bf18fa17: Time spent starting tablet: real 0.004s user 0.007s sys 0.000s
I20260927 02:37:12.900863 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:12.900630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.3:41113)
0927 02:37:12.900734 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:12.900792 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 02:37:12.901298 17145 tablet_copy_service.cc:342] P fb9f7e48bd2a42b0ad0cee0ee135f636: Request end of tablet copy session 060b505faf464a789ccc22f1bf18fa17-711abfa18eee412a84905888b8994321 received from {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51519
I20260927 02:37:12.901387 17145 tablet_copy_service.cc:433] P fb9f7e48bd2a42b0ad0cee0ee135f636: ending tablet copy session 060b505faf464a789ccc22f1bf18fa17-711abfa18eee412a84905888b8994321 on tablet 711abfa18eee412a84905888b8994321 with peer 060b505faf464a789ccc22f1bf18fa17
I20260927 02:37:12.901453 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:12.901251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.3:59681)
0927 02:37:12.901346 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:12.901391 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 02:37:13.330231 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:13.329934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.2:36919)
0927 02:37:13.330099 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:13.330148 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":52,"threads_started":1}
I20260927 02:37:13.330379 17259 raft_consensus.cc:1203] T 711abfa18eee412a84905888b8994321 P 060b505faf464a789ccc22f1bf18fa17 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260927 02:37:13.331182 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:13.330986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.3:43887)
0927 02:37:13.331076 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:13.331124 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 02:37:13.397503 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:13.397131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.1:49157)
0927 02:37:13.397343 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:13.397404 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":65,"threads_started":1}
I20260927 02:37:14.331297 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:14.330913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.2:59673)
0927 02:37:14.331157 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:14.331207 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":87,"threads_started":1}
I20260927 02:37:14.332150 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:14.331757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.3:38439)
0927 02:37:14.332016 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:14.332068 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":68,"threads_started":1}
I20260927 02:37:14.398509 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:14.398151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.1:44059)
0927 02:37:14.398374 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:14.398425 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":69,"threads_started":1}
I20260927 02:37:15.332331 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:15.331943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.2:48583)
0927 02:37:15.332169 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:15.332223 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":78,"threads_started":1}
I20260927 02:37:15.333107 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:15.332748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.3:57441)
0927 02:37:15.332980 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:15.333030 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":58,"threads_started":1}
I20260927 02:37:15.399607 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:15.399244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.1:41907)
0927 02:37:15.399465 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:15.399518 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":87,"threads_started":1}
I20260927 02:37:16.333570 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:16.333204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.2:40495)
0927 02:37:16.333427 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:16.333480 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":77,"threads_started":1}
I20260927 02:37:16.334187 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:16.333837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.3:57031)
0927 02:37:16.334048 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:16.334105 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":69,"threads_started":1}
I20260927 02:37:16.334569 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:16.334380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.2:44975)
0927 02:37:16.334477 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:16.334517 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 02:37:16.400806 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:16.400323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.1:34635)
0927 02:37:16.400644 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:16.400713 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":85,"threads_started":1}
I20260927 02:37:17.335490 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:17.334935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.3:42113)
0927 02:37:17.335277 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:17.335376 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":77,"threads_started":1}
I20260927 02:37:17.335528 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:17.335179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.2:33743)
0927 02:37:17.335403 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:17.335452 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":55,"threads_started":1}
I20260927 02:37:17.401865 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:17.401473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.1:58153)
0927 02:37:17.401723 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:17.401773 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":94,"threads_started":1}
I20260927 02:37:18.336488 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:18.336105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.3:34307)
0927 02:37:18.336340 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:18.336396 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":70,"threads_started":1}
I20260927 02:37:18.336535 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:18.336141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.2:36463)
0927 02:37:18.336412 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:18.336457 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":90,"threads_started":1}
I20260927 02:37:18.402875 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:18.402489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.1:55705)
0927 02:37:18.402728 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:18.402779 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":77,"threads_started":1}
I20260927 02:37:19.337555 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:19.337152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.3:56507)
0927 02:37:19.337379 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:19.337441 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":67,"threads_started":1}
I20260927 02:37:19.337903 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:19.337513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.2:41145)
0927 02:37:19.337749 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:19.337807 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":58,"threads_started":1}
I20260927 02:37:19.403880 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:19.403519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.1:58117)
0927 02:37:19.403741 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:19.403800 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":74,"threads_started":1}
I20260927 02:37:20.338765 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:20.338332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.3:43747)
0927 02:37:20.338588 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:20.338652 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":81,"threads_started":1}
I20260927 02:37:20.338907 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:20.338486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.2:41191)
0927 02:37:20.338775 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:20.338825 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":87,"threads_started":1}
I20260927 02:37:20.404965 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:20.404566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.1:36329)
0927 02:37:20.404798 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:20.404847 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":76,"threads_started":1}
I20260927 02:37:21.340071 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:21.339654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.3:53909)
0927 02:37:21.339899 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:21.339975 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":84,"threads_started":1}
I20260927 02:37:21.340247 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:21.339689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.2:45445)
0927 02:37:21.339997 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:21.340054 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":71,"threads_started":1}
I20260927 02:37:21.405881 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:21.405543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.1:58163)
0927 02:37:21.405756 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:21.405801 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":66,"threads_started":1}
I20260927 02:37:22.341271 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:22.340857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.3:42317)
0927 02:37:22.341094 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:22.341162 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":95,"threads_started":1}
I20260927 02:37:22.341318 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:22.340910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.2:53993)
0927 02:37:22.341189 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:22.341241 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":63,"threads_started":1}
I20260927 02:37:22.406901 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:22.406479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.1:55019)
0927 02:37:22.406747 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:22.406809 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":85,"threads_started":1}
I20260927 02:37:23.342329 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:23.341911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.3:59111)
0927 02:37:23.342161 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:23.342218 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":67,"threads_started":1}
I20260927 02:37:23.342342 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:23.341986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.2:50849)
0927 02:37:23.342214 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:23.342264 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":63,"threads_started":1}
I20260927 02:37:23.407853 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:23.407513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.1:43359)
0927 02:37:23.407720 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:23.407770 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":65,"threads_started":1}
I20260927 02:37:24.343545 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:24.343106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.2:40679)
0927 02:37:24.343394 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:24.343456 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":98,"threads_started":1}
I20260927 02:37:24.343595 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:24.343106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.3:51605)
0927 02:37:24.343450 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:24.343504 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":86,"threads_started":1}
I20260927 02:37:24.409041 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:24.408583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.1:41315)
0927 02:37:24.408860 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:24.408916 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":83,"threads_started":1}
I20260927 02:37:25.344769 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:25.344350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.3:51405)
0927 02:37:25.344624 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:25.344677 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":92,"threads_started":1}
I20260927 02:37:25.344772 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:25.344349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.2:52501)
0927 02:37:25.344615 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:25.344679 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":87,"threads_started":1}
I20260927 02:37:25.410336 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:25.409911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39525 (local address 127.0.98.1:38947)
0927 02:37:25.410161 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:25.410215 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":97,"threads_started":1}
I20260927 02:37:25.768121 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 16873
I20260927 02:37:25.773039 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 17039
I20260927 02:37:25.778282 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 17174
2026-09-27T02:37:25Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59037 ms)
[----------] 30 tests from SecurityITest (234837 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:37:25 dist-test-slave-grlk krb5kdc[17374](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:37:25 dist-test-slave-grlk krb5kdc[17374](info): set up 2 sockets
Sep 27 02:37:25 dist-test-slave-grlk krb5kdc[17374](info): commencing operation
krb5kdc: starting...
W20260927 02:37:26.740470 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/krb5kdc/test-user.keytab.
Sep 27 02:37:26 dist-test-slave-grlk krb5kdc[17374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:37:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:37:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:37:26.855207 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45537
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45537
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:37:26.961375 17387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:37:26.961622 17387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:37:26.961678 17387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:37:26.965633 17387 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:37:26.965680 17387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:37:26.965705 17387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:37:26.965728 17387 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:37:26.965757 17387 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:37:26.970767 17387 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:37:26.971673 17387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:37:26.972582 17387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:37:26.980479 17392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:26.980587 17393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:26.980644 17387 server_base.cc:1034] running on GCE node
W20260927 02:37:26.980489 17395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:26.981310 17387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:37:26.982278 17387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:37:26.983438 17387 hybrid_clock.cc:648] HybridClock initialized: now 1790476646983417 us; error 30 us; skew 500 ppm
Sep 27 02:37:26 dist-test-slave-grlk krb5kdc[17374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:37:26.985984 17387 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:37:26.986761 17387 webserver.cc:460] Webserver started at http://127.0.98.62:43601/ using document root <none> and password file <none>
I20260927 02:37:26.987252 17387 fs_manager.cc:359] Metadata directory not provided
I20260927 02:37:26.987313 17387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:37:26.987480 17387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:37:26.989383 17387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "e8d98b52f5db40fe8eadd23de6ed6080"
format_stamp: "Formatted at 2026-09-27 02:37:26 on dist-test-slave-grlk"
server_key: "bc93f32090d0a21a73370ed59ec4c49a"
server_key_iv: "2bbcaa8f301a94b3081e5b4b956d79d7"
server_key_version: "encryptionkey@0"
I20260927 02:37:26.989889 17387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "e8d98b52f5db40fe8eadd23de6ed6080"
format_stamp: "Formatted at 2026-09-27 02:37:26 on dist-test-slave-grlk"
server_key: "bc93f32090d0a21a73370ed59ec4c49a"
server_key_iv: "2bbcaa8f301a94b3081e5b4b956d79d7"
server_key_version: "encryptionkey@0"
I20260927 02:37:26.993407 17387 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:37:26.995821 17402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:26.997126 17387 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:37:26.997234 17387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "e8d98b52f5db40fe8eadd23de6ed6080"
format_stamp: "Formatted at 2026-09-27 02:37:26 on dist-test-slave-grlk"
server_key: "bc93f32090d0a21a73370ed59ec4c49a"
server_key_iv: "2bbcaa8f301a94b3081e5b4b956d79d7"
server_key_version: "encryptionkey@0"
I20260927 02:37:26.997331 17387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:37:27.012969 17387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:37:27.015954 17387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:37:27.016126 17387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:37:27.022528 17387 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45537
I20260927 02:37:27.022590 17454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45537 every 8 connection(s)
I20260927 02:37:27.023411 17387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:37:27.026005 17455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:37:27.030952 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 17387
I20260927 02:37:27.031066 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:37:27.031034 17455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8d98b52f5db40fe8eadd23de6ed6080: Bootstrap starting.
I20260927 02:37:27.031301 392 external_mini_cluster.cc:1428] Setting key 96b9d90abafa8830591d24ffb4eeeeb0
I20260927 02:37:27.033470 17455 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8d98b52f5db40fe8eadd23de6ed6080: Neither blocks nor log segments found. Creating new log.
I20260927 02:37:27.034297 17455 log.cc:824] T 00000000000000000000000000000000 P e8d98b52f5db40fe8eadd23de6ed6080: Log is configured to *not* fsync() on all Append() calls
I20260927 02:37:27.036345 17455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8d98b52f5db40fe8eadd23de6ed6080: No bootstrap required, opened a new log
I20260927 02:37:27.040015 17455 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8d98b52f5db40fe8eadd23de6ed6080 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8d98b52f5db40fe8eadd23de6ed6080" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45537 } }
I20260927 02:37:27.040234 17455 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8d98b52f5db40fe8eadd23de6ed6080 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:37:27.040282 17455 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8d98b52f5db40fe8eadd23de6ed6080 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8d98b52f5db40fe8eadd23de6ed6080, State: Initialized, Role: FOLLOWER
Sep 27 02:37:27 dist-test-slave-grlk krb5kdc[17374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:37:27.040747 17455 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8d98b52f5db40fe8eadd23de6ed6080 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8d98b52f5db40fe8eadd23de6ed6080" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45537 } }
I20260927 02:37:27.040923 17455 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8d98b52f5db40fe8eadd23de6ed6080 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:37:27.041061 17455 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8d98b52f5db40fe8eadd23de6ed6080 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:37:27.041167 17455 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8d98b52f5db40fe8eadd23de6ed6080 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:37:27.042189 17455 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8d98b52f5db40fe8eadd23de6ed6080 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8d98b52f5db40fe8eadd23de6ed6080" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45537 } }
I20260927 02:37:27.042505 17455 leader_election.cc:302] T 00000000000000000000000000000000 P e8d98b52f5db40fe8eadd23de6ed6080 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8d98b52f5db40fe8eadd23de6ed6080; no voters:
I20260927 02:37:27.042779 17455 leader_election.cc:288] T 00000000000000000000000000000000 P e8d98b52f5db40fe8eadd23de6ed6080 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:37:27.042873 17460 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8d98b52f5db40fe8eadd23de6ed6080 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:37:27.043052 17460 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8d98b52f5db40fe8eadd23de6ed6080 [term 1 LEADER]: Becoming Leader. State: Replica: e8d98b52f5db40fe8eadd23de6ed6080, State: Running, Role: LEADER
I20260927 02:37:27.043399 17460 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8d98b52f5db40fe8eadd23de6ed6080 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8d98b52f5db40fe8eadd23de6ed6080" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45537 } }
I20260927 02:37:27.044029 17455 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8d98b52f5db40fe8eadd23de6ed6080 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:37:27.044509 17460 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8d98b52f5db40fe8eadd23de6ed6080 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8d98b52f5db40fe8eadd23de6ed6080. Latest consensus state: current_term: 1 leader_uuid: "e8d98b52f5db40fe8eadd23de6ed6080" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8d98b52f5db40fe8eadd23de6ed6080" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45537 } } }
I20260927 02:37:27.045298 17462 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8d98b52f5db40fe8eadd23de6ed6080 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8d98b52f5db40fe8eadd23de6ed6080" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8d98b52f5db40fe8eadd23de6ed6080" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45537 } } }
I20260927 02:37:27.045423 17460 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8d98b52f5db40fe8eadd23de6ed6080 [sys.catalog]: This master's current role is: LEADER
I20260927 02:37:27.045495 17462 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8d98b52f5db40fe8eadd23de6ed6080 [sys.catalog]: This master's current role is: LEADER
I20260927 02:37:27.045742 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:27.032497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54798 (local address 127.0.98.62:45537)
0927 02:37:27.032874 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:27.032891 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:27.032954 (+ 63us) server_negotiation.cc:407] Connection header received
0927 02:37:27.033741 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:27.033763 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:27.034145 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:27.034516 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:27.035252 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:27.036663 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:27.037614 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:27.038301 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:27.038467 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:27.041099 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:27.041104 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:27.041125 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:37:27.041159 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:27.042907 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:27.043351 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:27.043357 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:27.043363 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:27.043469 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:27.043752 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:27.043755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:27.043757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:27.044148 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:27.044336 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:27.044570 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 02:37:27.044874 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":92,"threads_started":1}
I20260927 02:37:27.045841 17471 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:37:27.048693 17471 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:37:27.053848 17471 catalog_manager.cc:1347] Generated new cluster ID: 5cd63ca25b764575bcefd2adf70a54ae
I20260927 02:37:27.053912 17471 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:37:27.061936 17471 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:37:27.062819 17471 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:37:27.072496 17471 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8d98b52f5db40fe8eadd23de6ed6080: Generated new TSK 0
I20260927 02:37:27.073107 17471 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:37:27.101447 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45537
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:37:27.207358 17481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:37:27.207595 17481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:37:27.207661 17481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:37:27.211544 17481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:37:27.211592 17481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:37:27.211675 17481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:37:27.216722 17481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:37:27.217630 17481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:37:27.218530 17481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:37:27.227329 17486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:27.227396 17487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:27.227465 17481 server_base.cc:1034] running on GCE node
W20260927 02:37:27.227386 17489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:27.227914 17481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:37:27.228470 17481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:37:27.229614 17481 hybrid_clock.cc:648] HybridClock initialized: now 1790476647229594 us; error 32 us; skew 500 ppm
Sep 27 02:37:27 dist-test-slave-grlk krb5kdc[17374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:37:27.232034 17481 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:37:27.232774 17481 webserver.cc:460] Webserver started at http://127.0.98.1:37551/ using document root <none> and password file <none>
I20260927 02:37:27.233285 17481 fs_manager.cc:359] Metadata directory not provided
I20260927 02:37:27.233349 17481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:37:27.233515 17481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:37:27.235355 17481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-0/data/instance:
uuid: "1d132e75fb7746eea86aa8f7af4603cf"
format_stamp: "Formatted at 2026-09-27 02:37:27 on dist-test-slave-grlk"
server_key: "c8b17383293ad152a4fa7e08a11011ed"
server_key_iv: "e83c16a0b7f3de9d31e2778c5fe53314"
server_key_version: "encryptionkey@0"
I20260927 02:37:27.235863 17481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1d132e75fb7746eea86aa8f7af4603cf"
format_stamp: "Formatted at 2026-09-27 02:37:27 on dist-test-slave-grlk"
server_key: "c8b17383293ad152a4fa7e08a11011ed"
server_key_iv: "e83c16a0b7f3de9d31e2778c5fe53314"
server_key_version: "encryptionkey@0"
I20260927 02:37:27.239145 17481 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:37:27.241317 17496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:27.242417 17481 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:37:27.242514 17481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-0/wal
uuid: "1d132e75fb7746eea86aa8f7af4603cf"
format_stamp: "Formatted at 2026-09-27 02:37:27 on dist-test-slave-grlk"
server_key: "c8b17383293ad152a4fa7e08a11011ed"
server_key_iv: "e83c16a0b7f3de9d31e2778c5fe53314"
server_key_version: "encryptionkey@0"
I20260927 02:37:27.242597 17481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:37:27.264683 17481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:37:27.267693 17481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:37:27.267881 17481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:37:27.268431 17481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:37:27.269179 17481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:37:27.269222 17481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:27.269258 17481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:37:27.269279 17481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:27.276888 17481 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38073
I20260927 02:37:27.276921 17609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38073 every 8 connection(s)
I20260927 02:37:27.277685 17481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 02:37:27 dist-test-slave-grlk krb5kdc[17374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:37:27.286583 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 17481
I20260927 02:37:27.286697 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:37:27.286979 392 external_mini_cluster.cc:1428] Setting key e29b59a90310fb788ed054228b3a3bc7
I20260927 02:37:27.290150 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:27.279282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33121 (local address 127.0.98.62:45537)
0927 02:37:27.279399 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:27.279403 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:27.280122 (+ 719us) server_negotiation.cc:407] Connection header received
0927 02:37:27.281340 (+ 1218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:27.281343 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:27.281397 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:27.281471 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:27.282520 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:27.283041 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:27.284485 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:27.284714 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:27.284803 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:27.286969 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:27.286973 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:27.286976 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:37:27.287003 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:27.288285 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:27.288788 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:27.288792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:27.288794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:27.288879 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:27.289199 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:27.289202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:27.289203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:27.289410 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:27.289496 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:27.289880 (+ 384us) server_negotiation.cc:299] Negotiation successful
0927 02:37:27.289999 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 02:37:27.291041 17612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:27.279529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45537 (local address 127.0.98.1:33121)
0927 02:37:27.279971 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:27.280009 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:27.281141 (+ 1132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:27.281629 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:27.281635 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:27.281953 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:37:27.282382 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:27.282395 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:27.283155 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:27.283158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:27.284373 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:27.284383 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:27.284846 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:27.284850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:27.285046 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:27.285484 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:37:27.285499 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:37:27.286819 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:37:27.288401 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:37:27.288407 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:37:27.288420 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:37:27.288695 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:37:27.288994 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:37:27.288996 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:37:27.288997 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:37:27.289122 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:37:27.289490 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:37:27.289495 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:37:27.289689 (+ 194us) client_negotiation.cc:769] Sending connection context
0927 02:37:27.289840 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 02:37:27.290148 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":113,"threads_started":1}
I20260927 02:37:27.292109 17610 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45537
I20260927 02:37:27.292330 17610 heartbeater.cc:464] Registering TS with master...
I20260927 02:37:27.292755 17610 heartbeater.cc:511] Master 127.0.98.62:45537 requested a full tablet report, sending...
I20260927 02:37:27.294330 17419 ts_manager.cc:194] Registered new tserver with Master: 1d132e75fb7746eea86aa8f7af4603cf (127.0.98.1:38073)
I20260927 02:37:27.295819 17419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33121
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:37:27.310945 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45537
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:37:27.413834 17615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:37:27.414072 17615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:37:27.414112 17615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:37:27.418104 17615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:37:27.418159 17615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:37:27.418215 17615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:37:27.423236 17615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:37:27.424129 17615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:37:27.425052 17615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:37:27.433673 17620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:27.433765 17615 server_base.cc:1034] running on GCE node
W20260927 02:37:27.433771 17621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:27.433979 17623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:27.434327 17615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:37:27.434963 17615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:37:27.436172 17615 hybrid_clock.cc:648] HybridClock initialized: now 1790476647436132 us; error 53 us; skew 500 ppm
Sep 27 02:37:27 dist-test-slave-grlk krb5kdc[17374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:37:27.438619 17615 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:37:27.439378 17615 webserver.cc:460] Webserver started at http://127.0.98.2:38527/ using document root <none> and password file <none>
I20260927 02:37:27.439864 17615 fs_manager.cc:359] Metadata directory not provided
I20260927 02:37:27.439925 17615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:37:27.440094 17615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:37:27.441975 17615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-1/data/instance:
uuid: "5298b8a3b7924ad7beefc0cea12d7352"
format_stamp: "Formatted at 2026-09-27 02:37:27 on dist-test-slave-grlk"
server_key: "1e8c77e03d7a30486f649373732cfe1d"
server_key_iv: "82dd83e2549038e361065f9fbd5bb168"
server_key_version: "encryptionkey@0"
I20260927 02:37:27.442468 17615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5298b8a3b7924ad7beefc0cea12d7352"
format_stamp: "Formatted at 2026-09-27 02:37:27 on dist-test-slave-grlk"
server_key: "1e8c77e03d7a30486f649373732cfe1d"
server_key_iv: "82dd83e2549038e361065f9fbd5bb168"
server_key_version: "encryptionkey@0"
I20260927 02:37:27.446019 17615 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:37:27.448175 17630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:27.449265 17615 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:37:27.449368 17615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-1/wal
uuid: "5298b8a3b7924ad7beefc0cea12d7352"
format_stamp: "Formatted at 2026-09-27 02:37:27 on dist-test-slave-grlk"
server_key: "1e8c77e03d7a30486f649373732cfe1d"
server_key_iv: "82dd83e2549038e361065f9fbd5bb168"
server_key_version: "encryptionkey@0"
I20260927 02:37:27.449447 17615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:37:27.471580 17615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:37:27.473923 17615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:37:27.474068 17615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:37:27.474560 17615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:37:27.475247 17615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:37:27.475291 17615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:27.475347 17615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:37:27.475373 17615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:27.482926 17615 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41557
I20260927 02:37:27.482949 17743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41557 every 8 connection(s)
I20260927 02:37:27.483670 17615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:37:27.485980 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 17615
I20260927 02:37:27.486104 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:37:27.486366 392 external_mini_cluster.cc:1428] Setting key 34a65dca17501a62454eb9595906d437
Sep 27 02:37:27 dist-test-slave-grlk krb5kdc[17374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:37:27.496500 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:27.485184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49319 (local address 127.0.98.62:45537)
0927 02:37:27.485308 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:27.485312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:27.486065 (+ 753us) server_negotiation.cc:407] Connection header received
0927 02:37:27.487324 (+ 1259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:27.487327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:27.487367 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:27.487425 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:27.488745 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:27.489306 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:27.490803 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:27.491012 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:27.491078 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:27.493505 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:27.493508 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:27.493510 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:37:27.493532 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:27.494772 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:27.495260 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:27.495263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:27.495265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:27.495349 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:27.495647 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:27.495648 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:27.495650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:27.495820 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:27.495889 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:27.496292 (+ 403us) server_negotiation.cc:299] Negotiation successful
0927 02:37:27.496383 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 02:37:27.497249 17746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:27.485534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45537 (local address 127.0.98.2:49319)
0927 02:37:27.485930 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:27.485966 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:27.487125 (+ 1159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:27.487589 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:27.487595 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:27.488008 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:37:27.488605 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:27.488623 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:27.489414 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:27.489417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:27.490693 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:27.490704 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:27.491134 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:27.491137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:27.491327 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:27.491845 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:37:27.491863 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:37:27.493361 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:37:27.494882 (+ 1521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:37:27.494887 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:37:27.494900 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:37:27.495166 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:37:27.495443 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:37:27.495446 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:37:27.495447 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:37:27.495573 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:37:27.495893 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:37:27.495898 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:37:27.496113 (+ 215us) client_negotiation.cc:769] Sending connection context
0927 02:37:27.496263 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 02:37:27.496491 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":106,"threads_started":1}
I20260927 02:37:27.498270 17744 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45537
I20260927 02:37:27.498584 17744 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 02:37:27.499063 17744 heartbeater.cc:511] Master 127.0.98.62:45537 requested a full tablet report, sending...
I20260927 02:37:27.500016 17419 ts_manager.cc:194] Registered new tserver with Master: 5298b8a3b7924ad7beefc0cea12d7352 (127.0.98.2:41557)
I20260927 02:37:27.500543 17419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49319
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:37:27.511382 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45537
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:37:27.612489 17749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:37:27.612700 17749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:37:27.612753 17749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:37:27.616627 17749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:37:27.616679 17749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:37:27.616755 17749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:37:27.621634 17749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:37:27.622476 17749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:37:27.623347 17749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:37:27.631850 17755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:27.631928 17749 server_base.cc:1034] running on GCE node
W20260927 02:37:27.631829 17757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:27.631934 17754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:27.632405 17749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:37:27.633015 17749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:37:27.634166 17749 hybrid_clock.cc:648] HybridClock initialized: now 1790476647634154 us; error 25 us; skew 500 ppm
Sep 27 02:37:27 dist-test-slave-grlk krb5kdc[17374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:37:27.636557 17749 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:37:27.637364 17749 webserver.cc:460] Webserver started at http://127.0.98.3:41261/ using document root <none> and password file <none>
I20260927 02:37:27.637861 17749 fs_manager.cc:359] Metadata directory not provided
I20260927 02:37:27.637921 17749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:37:27.638088 17749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:37:27.639933 17749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-2/data/instance:
uuid: "8e6c7735499b4daa9bfe9811b07dcde5"
format_stamp: "Formatted at 2026-09-27 02:37:27 on dist-test-slave-grlk"
server_key: "4105cba6a698da37204b7d8fe05fc0cb"
server_key_iv: "66e4e4531a515d56e875661936bb2886"
server_key_version: "encryptionkey@0"
I20260927 02:37:27.640420 17749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8e6c7735499b4daa9bfe9811b07dcde5"
format_stamp: "Formatted at 2026-09-27 02:37:27 on dist-test-slave-grlk"
server_key: "4105cba6a698da37204b7d8fe05fc0cb"
server_key_iv: "66e4e4531a515d56e875661936bb2886"
server_key_version: "encryptionkey@0"
I20260927 02:37:27.643661 17749 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 02:37:27.645941 17764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:27.647003 17749 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:37:27.647100 17749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-2/wal
uuid: "8e6c7735499b4daa9bfe9811b07dcde5"
format_stamp: "Formatted at 2026-09-27 02:37:27 on dist-test-slave-grlk"
server_key: "4105cba6a698da37204b7d8fe05fc0cb"
server_key_iv: "66e4e4531a515d56e875661936bb2886"
server_key_version: "encryptionkey@0"
I20260927 02:37:27.647181 17749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:37:27.665186 17749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:37:27.667722 17749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:37:27.667884 17749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:37:27.668370 17749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:37:27.669119 17749 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:37:27.669165 17749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:27.669214 17749 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:37:27.669243 17749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:27.677114 17749 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33715
I20260927 02:37:27.677287 17877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33715 every 8 connection(s)
I20260927 02:37:27.677937 17749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:37:27.685925 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 17749
I20260927 02:37:27.686054 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:37:27.686344 392 external_mini_cluster.cc:1428] Setting key 6b2fe18c8cb2f01d0a6157a5ca75eae1
Sep 27 02:37:27 dist-test-slave-grlk krb5kdc[17374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:37:27.690591 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:27.679598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34941 (local address 127.0.98.62:45537)
0927 02:37:27.679723 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:27.679726 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:27.680467 (+ 741us) server_negotiation.cc:407] Connection header received
0927 02:37:27.681680 (+ 1213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:27.681683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:27.681738 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:27.681819 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:27.682969 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:27.683487 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:27.684584 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:27.684797 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:27.684890 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:27.687452 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:27.687456 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:27.687459 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:37:27.687487 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:27.688794 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:27.689289 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:27.689291 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:27.689293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:27.689364 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:27.689719 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:27.689721 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:27.689722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:27.689898 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:27.689963 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:27.690386 (+ 423us) server_negotiation.cc:299] Negotiation successful
0927 02:37:27.690478 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 02:37:27.691380 17880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:27.679925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45537 (local address 127.0.98.3:34941)
0927 02:37:27.680332 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:27.680369 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:27.681483 (+ 1114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:27.681977 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:27.681984 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:27.682324 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:37:27.682825 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:27.682839 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:27.683599 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:27.683602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:27.684465 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:27.684475 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:27.684961 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:27.684967 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:27.685118 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:27.685638 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:37:27.685653 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:37:27.687310 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:37:27.688913 (+ 1603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:37:27.688918 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:37:27.688953 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:37:27.689206 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:37:27.689464 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:37:27.689467 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:37:27.689468 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:37:27.689614 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:37:27.689968 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:37:27.689972 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:37:27.690204 (+ 232us) client_negotiation.cc:769] Sending connection context
0927 02:37:27.690348 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 02:37:27.690588 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":113,"threads_started":1}
I20260927 02:37:27.692358 17878 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45537
I20260927 02:37:27.692548 17878 heartbeater.cc:464] Registering TS with master...
I20260927 02:37:27.692975 17878 heartbeater.cc:511] Master 127.0.98.62:45537 requested a full tablet report, sending...
I20260927 02:37:27.693853 17419 ts_manager.cc:194] Registered new tserver with Master: 8e6c7735499b4daa9bfe9811b07dcde5 (127.0.98.3:33715)
I20260927 02:37:27.694391 17419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34941
I20260927 02:37:27.699748 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:37:27.707491 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:27.701481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54806 (local address 127.0.98.62:45537)
0927 02:37:27.701576 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:27.701579 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:27.701717 (+ 138us) server_negotiation.cc:407] Connection header received
0927 02:37:27.701830 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:27.701833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:27.701872 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:27.701932 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:27.702493 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:27.702965 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:27.703684 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:27.703876 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:27.703937 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:27.705010 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:27.705013 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:27.705015 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:37:27.705033 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:27.706238 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:27.706556 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:27.706558 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:27.706559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:27.706619 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:27.706795 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:27.706797 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:27.706798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:27.706982 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:27.707078 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:27.707302 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 02:37:27.707384 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 02:37:27.709961 17419 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54806:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:37:27.712724 17419 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:37:27.725342 17890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:27.720488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33116 (local address 127.0.98.2:41557)
0927 02:37:27.720864 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:27.720876 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:27.720897 (+ 21us) server_negotiation.cc:407] Connection header received
0927 02:37:27.720989 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:27.720994 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:27.721189 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:27.721353 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:37:27.721813 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:27.722863 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:27.724129 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:27.724700 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:27.724793 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:27.724965 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:27.725118 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 02:37:27.725206 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":84,"threads_started":1}
I20260927 02:37:27.725345 17889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:27.720308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41557 (local address 127.0.0.1:33116)
0927 02:37:27.720614 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:27.720661 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:27.720777 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:27.721538 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:27.721544 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:27.721568 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:37:27.721692 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:27.721700 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:27.722973 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:27.722976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:27.724003 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:27.724012 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:27.724805 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:27.724809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:27.724912 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:27.724986 (+ 74us) client_negotiation.cc:769] Sending connection context
0927 02:37:27.725080 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 02:37:27.725193 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":51,"thread_start_us":77,"threads_started":1}
I20260927 02:37:27.726485 17894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:27.721505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38073 (local address 127.0.0.1:43890)
0927 02:37:27.721897 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:27.721909 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:27.721990 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:27.722340 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:27.722342 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:27.722357 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:37:27.722446 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:27.722452 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:27.723798 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:27.723802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:27.725108 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:27.725119 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:27.726148 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:27.726151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:27.726234 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:27.726256 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 02:37:27.726327 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 02:37:27.726389 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":33,"thread_start_us":74,"threads_started":1}
I20260927 02:37:27.726820 17892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:27.721074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41800 (local address 127.0.98.3:33715)
0927 02:37:27.721631 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:27.721642 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:27.721847 (+ 205us) server_negotiation.cc:407] Connection header received
0927 02:37:27.722038 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:27.722044 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:27.722220 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:27.722331 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:37:27.722700 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:27.723767 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:27.725239 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:27.726025 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:27.726133 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:27.726256 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:27.726575 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 02:37:27.726663 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":136,"threads_started":1}
I20260927 02:37:27.726944 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:27.721181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43890 (local address 127.0.98.1:38073)
0927 02:37:27.721852 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:27.721863 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:27.721985 (+ 122us) server_negotiation.cc:407] Connection header received
0927 02:37:27.722091 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:27.722095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:27.722226 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:27.722343 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:37:27.722549 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:27.723659 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:27.725239 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:27.726025 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:27.726133 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:27.726256 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:27.726718 (+ 462us) server_negotiation.cc:299] Negotiation successful
0927 02:37:27.726800 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":111,"threads_started":1}
I20260927 02:37:27.726712 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:27.720624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33715 (local address 127.0.0.1:41800)
0927 02:37:27.721757 (+ 1133us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:27.721769 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:27.721912 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:27.722501 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:27.722504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:27.722519 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:37:27.722596 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:27.722601 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:27.723888 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:27.723891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:27.725110 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:27.725119 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:27.726341 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:27.726345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:27.726430 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:27.726453 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 02:37:27.726525 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 02:37:27.726589 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1082,"mutex_wait_us":32,"thread_start_us":104,"threads_started":1}
I20260927 02:37:27.728266 17678 tablet_service.cc:1467] Processing CreateTablet for tablet 78e4e77bcefd45808527b93b7a7311fb (DEFAULT_TABLE table=test-table [id=2c82556f21dc4dc1bdbf7d66a934e5c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:37:27.729295 17678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78e4e77bcefd45808527b93b7a7311fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:37:27.729382 17812 tablet_service.cc:1467] Processing CreateTablet for tablet 78e4e77bcefd45808527b93b7a7311fb (DEFAULT_TABLE table=test-table [id=2c82556f21dc4dc1bdbf7d66a934e5c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:37:27.730373 17812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78e4e77bcefd45808527b93b7a7311fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:37:27.731174 17544 tablet_service.cc:1467] Processing CreateTablet for tablet 78e4e77bcefd45808527b93b7a7311fb (DEFAULT_TABLE table=test-table [id=2c82556f21dc4dc1bdbf7d66a934e5c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:37:27.732131 17544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78e4e77bcefd45808527b93b7a7311fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:37:27.735912 17895 tablet_bootstrap.cc:492] T 78e4e77bcefd45808527b93b7a7311fb P 5298b8a3b7924ad7beefc0cea12d7352: Bootstrap starting.
I20260927 02:37:27.736922 17896 tablet_bootstrap.cc:492] T 78e4e77bcefd45808527b93b7a7311fb P 8e6c7735499b4daa9bfe9811b07dcde5: Bootstrap starting.
I20260927 02:37:27.738500 17895 tablet_bootstrap.cc:654] T 78e4e77bcefd45808527b93b7a7311fb P 5298b8a3b7924ad7beefc0cea12d7352: Neither blocks nor log segments found. Creating new log.
I20260927 02:37:27.738492 17897 tablet_bootstrap.cc:492] T 78e4e77bcefd45808527b93b7a7311fb P 1d132e75fb7746eea86aa8f7af4603cf: Bootstrap starting.
I20260927 02:37:27.738974 17896 tablet_bootstrap.cc:654] T 78e4e77bcefd45808527b93b7a7311fb P 8e6c7735499b4daa9bfe9811b07dcde5: Neither blocks nor log segments found. Creating new log.
I20260927 02:37:27.739439 17895 log.cc:824] T 78e4e77bcefd45808527b93b7a7311fb P 5298b8a3b7924ad7beefc0cea12d7352: Log is configured to *not* fsync() on all Append() calls
I20260927 02:37:27.740093 17896 log.cc:824] T 78e4e77bcefd45808527b93b7a7311fb P 8e6c7735499b4daa9bfe9811b07dcde5: Log is configured to *not* fsync() on all Append() calls
I20260927 02:37:27.740200 17897 tablet_bootstrap.cc:654] T 78e4e77bcefd45808527b93b7a7311fb P 1d132e75fb7746eea86aa8f7af4603cf: Neither blocks nor log segments found. Creating new log.
I20260927 02:37:27.741312 17897 log.cc:824] T 78e4e77bcefd45808527b93b7a7311fb P 1d132e75fb7746eea86aa8f7af4603cf: Log is configured to *not* fsync() on all Append() calls
I20260927 02:37:27.741815 17895 tablet_bootstrap.cc:492] T 78e4e77bcefd45808527b93b7a7311fb P 5298b8a3b7924ad7beefc0cea12d7352: No bootstrap required, opened a new log
I20260927 02:37:27.742009 17895 ts_tablet_manager.cc:1397] T 78e4e77bcefd45808527b93b7a7311fb P 5298b8a3b7924ad7beefc0cea12d7352: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 02:37:27.742139 17896 tablet_bootstrap.cc:492] T 78e4e77bcefd45808527b93b7a7311fb P 8e6c7735499b4daa9bfe9811b07dcde5: No bootstrap required, opened a new log
I20260927 02:37:27.742309 17896 ts_tablet_manager.cc:1397] T 78e4e77bcefd45808527b93b7a7311fb P 8e6c7735499b4daa9bfe9811b07dcde5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 02:37:27.742857 17897 tablet_bootstrap.cc:492] T 78e4e77bcefd45808527b93b7a7311fb P 1d132e75fb7746eea86aa8f7af4603cf: No bootstrap required, opened a new log
I20260927 02:37:27.743013 17897 ts_tablet_manager.cc:1397] T 78e4e77bcefd45808527b93b7a7311fb P 1d132e75fb7746eea86aa8f7af4603cf: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 02:37:27.745939 17896 raft_consensus.cc:348] T 78e4e77bcefd45808527b93b7a7311fb P 8e6c7735499b4daa9bfe9811b07dcde5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5298b8a3b7924ad7beefc0cea12d7352" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41557 } } peers { permanent_uuid: "1d132e75fb7746eea86aa8f7af4603cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38073 } } peers { permanent_uuid: "8e6c7735499b4daa9bfe9811b07dcde5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33715 } }
I20260927 02:37:27.745939 17895 raft_consensus.cc:348] T 78e4e77bcefd45808527b93b7a7311fb P 5298b8a3b7924ad7beefc0cea12d7352 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5298b8a3b7924ad7beefc0cea12d7352" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41557 } } peers { permanent_uuid: "1d132e75fb7746eea86aa8f7af4603cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38073 } } peers { permanent_uuid: "8e6c7735499b4daa9bfe9811b07dcde5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33715 } }
I20260927 02:37:27.746228 17896 raft_consensus.cc:374] T 78e4e77bcefd45808527b93b7a7311fb P 8e6c7735499b4daa9bfe9811b07dcde5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:37:27.746228 17895 raft_consensus.cc:374] T 78e4e77bcefd45808527b93b7a7311fb P 5298b8a3b7924ad7beefc0cea12d7352 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:37:27.746110 17897 raft_consensus.cc:348] T 78e4e77bcefd45808527b93b7a7311fb P 1d132e75fb7746eea86aa8f7af4603cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5298b8a3b7924ad7beefc0cea12d7352" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41557 } } peers { permanent_uuid: "1d132e75fb7746eea86aa8f7af4603cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38073 } } peers { permanent_uuid: "8e6c7735499b4daa9bfe9811b07dcde5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33715 } }
I20260927 02:37:27.746294 17895 raft_consensus.cc:729] T 78e4e77bcefd45808527b93b7a7311fb P 5298b8a3b7924ad7beefc0cea12d7352 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5298b8a3b7924ad7beefc0cea12d7352, State: Initialized, Role: FOLLOWER
I20260927 02:37:27.746309 17896 raft_consensus.cc:729] T 78e4e77bcefd45808527b93b7a7311fb P 8e6c7735499b4daa9bfe9811b07dcde5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e6c7735499b4daa9bfe9811b07dcde5, State: Initialized, Role: FOLLOWER
I20260927 02:37:27.746363 17897 raft_consensus.cc:374] T 78e4e77bcefd45808527b93b7a7311fb P 1d132e75fb7746eea86aa8f7af4603cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:37:27.746423 17897 raft_consensus.cc:729] T 78e4e77bcefd45808527b93b7a7311fb P 1d132e75fb7746eea86aa8f7af4603cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d132e75fb7746eea86aa8f7af4603cf, State: Initialized, Role: FOLLOWER
I20260927 02:37:27.746779 17896 consensus_queue.cc:260] T 78e4e77bcefd45808527b93b7a7311fb P 8e6c7735499b4daa9bfe9811b07dcde5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5298b8a3b7924ad7beefc0cea12d7352" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41557 } } peers { permanent_uuid: "1d132e75fb7746eea86aa8f7af4603cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38073 } } peers { permanent_uuid: "8e6c7735499b4daa9bfe9811b07dcde5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33715 } }
I20260927 02:37:27.746771 17895 consensus_queue.cc:260] T 78e4e77bcefd45808527b93b7a7311fb P 5298b8a3b7924ad7beefc0cea12d7352 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5298b8a3b7924ad7beefc0cea12d7352" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41557 } } peers { permanent_uuid: "1d132e75fb7746eea86aa8f7af4603cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38073 } } peers { permanent_uuid: "8e6c7735499b4daa9bfe9811b07dcde5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33715 } }
I20260927 02:37:27.746856 17897 consensus_queue.cc:260] T 78e4e77bcefd45808527b93b7a7311fb P 1d132e75fb7746eea86aa8f7af4603cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5298b8a3b7924ad7beefc0cea12d7352" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41557 } } peers { permanent_uuid: "1d132e75fb7746eea86aa8f7af4603cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38073 } } peers { permanent_uuid: "8e6c7735499b4daa9bfe9811b07dcde5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33715 } }
I20260927 02:37:27.747483 17610 heartbeater.cc:503] Master 127.0.98.62:45537 was elected leader, sending a full tablet report...
I20260927 02:37:27.747774 17896 ts_tablet_manager.cc:1428] T 78e4e77bcefd45808527b93b7a7311fb P 8e6c7735499b4daa9bfe9811b07dcde5: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 02:37:27.747819 17895 ts_tablet_manager.cc:1428] T 78e4e77bcefd45808527b93b7a7311fb P 5298b8a3b7924ad7beefc0cea12d7352: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 02:37:27.748274 17897 ts_tablet_manager.cc:1428] T 78e4e77bcefd45808527b93b7a7311fb P 1d132e75fb7746eea86aa8f7af4603cf: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 02:37:27.748339 17878 heartbeater.cc:503] Master 127.0.98.62:45537 was elected leader, sending a full tablet report...
I20260927 02:37:27.749199 17744 heartbeater.cc:503] Master 127.0.98.62:45537 was elected leader, sending a full tablet report...
W20260927 02:37:27.778700 17611 tablet.cc:2369] T 78e4e77bcefd45808527b93b7a7311fb P 1d132e75fb7746eea86aa8f7af4603cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:37:27.928864 17879 tablet.cc:2369] T 78e4e77bcefd45808527b93b7a7311fb P 8e6c7735499b4daa9bfe9811b07dcde5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:37:27.984700 17745 tablet.cc:2369] T 78e4e77bcefd45808527b93b7a7311fb P 5298b8a3b7924ad7beefc0cea12d7352: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:37:28.015614 17903 raft_consensus.cc:482] T 78e4e77bcefd45808527b93b7a7311fb P 1d132e75fb7746eea86aa8f7af4603cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:37:28.015812 17903 raft_consensus.cc:504] T 78e4e77bcefd45808527b93b7a7311fb P 1d132e75fb7746eea86aa8f7af4603cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5298b8a3b7924ad7beefc0cea12d7352" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41557 } } peers { permanent_uuid: "1d132e75fb7746eea86aa8f7af4603cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38073 } } peers { permanent_uuid: "8e6c7735499b4daa9bfe9811b07dcde5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33715 } }
I20260927 02:37:28.016896 17903 leader_election.cc:288] T 78e4e77bcefd45808527b93b7a7311fb P 1d132e75fb7746eea86aa8f7af4603cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5298b8a3b7924ad7beefc0cea12d7352 (127.0.98.2:41557), 8e6c7735499b4daa9bfe9811b07dcde5 (127.0.98.3:33715)
I20260927 02:37:28.020432 17890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:28.017236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49289 (local address 127.0.98.2:41557)
0927 02:37:28.017362 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:28.017366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:28.017525 (+ 159us) server_negotiation.cc:407] Connection header received
0927 02:37:28.017661 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:28.017663 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:28.017703 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:28.017762 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:37:28.018040 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:28.018521 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:28.019554 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:28.019930 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:28.020015 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:28.020074 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:28.020254 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 02:37:28.020315 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 02:37:28.020457 17905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:28.017142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41557 (local address 127.0.98.1:49289)
0927 02:37:28.017438 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:28.017450 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:28.017558 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:28.017781 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:28.017784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:28.017798 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:37:28.017944 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:28.017949 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:28.018615 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:28.018618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:28.019436 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:28.019445 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:28.020039 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:28.020042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:28.020142 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:28.020164 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 02:37:28.020233 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 02:37:28.020299 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":23,"spinlock_wait_cycles":16512,"thread_start_us":104,"threads_started":1}
I20260927 02:37:28.020521 17904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:28.017153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33715 (local address 127.0.98.1:55463)
0927 02:37:28.017461 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:28.017475 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:28.017574 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:28.017791 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:28.017794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:28.017808 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:37:28.017961 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:28.017967 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:28.018688 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:28.018691 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:28.019513 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:28.019522 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:28.020117 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:28.020120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:28.020232 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:28.020249 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 02:37:28.020309 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 02:37:28.020372 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":26,"thread_start_us":79,"threads_started":1}
I20260927 02:37:28.020505 17892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:28.017243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55463 (local address 127.0.98.3:33715)
0927 02:37:28.017463 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:28.017468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:28.017540 (+ 72us) server_negotiation.cc:407] Connection header received
0927 02:37:28.017666 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:28.017670 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:28.017712 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:28.017772 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:37:28.018058 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:28.018562 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:28.019622 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:28.020011 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:28.020099 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:28.020167 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:28.020334 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 02:37:28.020401 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260927 02:37:28.021034 17698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78e4e77bcefd45808527b93b7a7311fb" candidate_uuid: "1d132e75fb7746eea86aa8f7af4603cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5298b8a3b7924ad7beefc0cea12d7352" is_pre_election: true
I20260927 02:37:28.021103 17832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78e4e77bcefd45808527b93b7a7311fb" candidate_uuid: "1d132e75fb7746eea86aa8f7af4603cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e6c7735499b4daa9bfe9811b07dcde5" is_pre_election: true
I20260927 02:37:28.021338 17698 raft_consensus.cc:2454] T 78e4e77bcefd45808527b93b7a7311fb P 5298b8a3b7924ad7beefc0cea12d7352 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d132e75fb7746eea86aa8f7af4603cf in term 0.
I20260927 02:37:28.021338 17832 raft_consensus.cc:2454] T 78e4e77bcefd45808527b93b7a7311fb P 8e6c7735499b4daa9bfe9811b07dcde5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d132e75fb7746eea86aa8f7af4603cf in term 0.
I20260927 02:37:28.021798 17499 leader_election.cc:302] T 78e4e77bcefd45808527b93b7a7311fb P 1d132e75fb7746eea86aa8f7af4603cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d132e75fb7746eea86aa8f7af4603cf, 8e6c7735499b4daa9bfe9811b07dcde5; no voters:
I20260927 02:37:28.022028 17903 raft_consensus.cc:2781] T 78e4e77bcefd45808527b93b7a7311fb P 1d132e75fb7746eea86aa8f7af4603cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:37:28.022114 17903 raft_consensus.cc:482] T 78e4e77bcefd45808527b93b7a7311fb P 1d132e75fb7746eea86aa8f7af4603cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:37:28.022163 17903 raft_consensus.cc:3037] T 78e4e77bcefd45808527b93b7a7311fb P 1d132e75fb7746eea86aa8f7af4603cf [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:37:28.023124 17903 raft_consensus.cc:504] T 78e4e77bcefd45808527b93b7a7311fb P 1d132e75fb7746eea86aa8f7af4603cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5298b8a3b7924ad7beefc0cea12d7352" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41557 } } peers { permanent_uuid: "1d132e75fb7746eea86aa8f7af4603cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38073 } } peers { permanent_uuid: "8e6c7735499b4daa9bfe9811b07dcde5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33715 } }
I20260927 02:37:28.023454 17903 leader_election.cc:288] T 78e4e77bcefd45808527b93b7a7311fb P 1d132e75fb7746eea86aa8f7af4603cf [CANDIDATE]: Term 1 election: Requested vote from peers 5298b8a3b7924ad7beefc0cea12d7352 (127.0.98.2:41557), 8e6c7735499b4daa9bfe9811b07dcde5 (127.0.98.3:33715)
I20260927 02:37:28.023756 17698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78e4e77bcefd45808527b93b7a7311fb" candidate_uuid: "1d132e75fb7746eea86aa8f7af4603cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5298b8a3b7924ad7beefc0cea12d7352"
I20260927 02:37:28.023806 17832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78e4e77bcefd45808527b93b7a7311fb" candidate_uuid: "1d132e75fb7746eea86aa8f7af4603cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e6c7735499b4daa9bfe9811b07dcde5"
I20260927 02:37:28.023874 17698 raft_consensus.cc:3037] T 78e4e77bcefd45808527b93b7a7311fb P 5298b8a3b7924ad7beefc0cea12d7352 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:37:28.023900 17832 raft_consensus.cc:3037] T 78e4e77bcefd45808527b93b7a7311fb P 8e6c7735499b4daa9bfe9811b07dcde5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:37:28.025032 17698 raft_consensus.cc:2454] T 78e4e77bcefd45808527b93b7a7311fb P 5298b8a3b7924ad7beefc0cea12d7352 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d132e75fb7746eea86aa8f7af4603cf in term 1.
I20260927 02:37:28.025032 17832 raft_consensus.cc:2454] T 78e4e77bcefd45808527b93b7a7311fb P 8e6c7735499b4daa9bfe9811b07dcde5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d132e75fb7746eea86aa8f7af4603cf in term 1.
I20260927 02:37:28.025295 17499 leader_election.cc:302] T 78e4e77bcefd45808527b93b7a7311fb P 1d132e75fb7746eea86aa8f7af4603cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d132e75fb7746eea86aa8f7af4603cf, 8e6c7735499b4daa9bfe9811b07dcde5; no voters:
I20260927 02:37:28.025441 17903 raft_consensus.cc:2781] T 78e4e77bcefd45808527b93b7a7311fb P 1d132e75fb7746eea86aa8f7af4603cf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:37:28.025631 17903 raft_consensus.cc:686] T 78e4e77bcefd45808527b93b7a7311fb P 1d132e75fb7746eea86aa8f7af4603cf [term 1 LEADER]: Becoming Leader. State: Replica: 1d132e75fb7746eea86aa8f7af4603cf, State: Running, Role: LEADER
I20260927 02:37:28.025942 17903 consensus_queue.cc:237] T 78e4e77bcefd45808527b93b7a7311fb P 1d132e75fb7746eea86aa8f7af4603cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5298b8a3b7924ad7beefc0cea12d7352" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41557 } } peers { permanent_uuid: "1d132e75fb7746eea86aa8f7af4603cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38073 } } peers { permanent_uuid: "8e6c7735499b4daa9bfe9811b07dcde5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33715 } }
I20260927 02:37:28.029145 17418 catalog_manager.cc:5495] T 78e4e77bcefd45808527b93b7a7311fb P 1d132e75fb7746eea86aa8f7af4603cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d132e75fb7746eea86aa8f7af4603cf (127.0.98.1). New cstate: current_term: 1 leader_uuid: "1d132e75fb7746eea86aa8f7af4603cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5298b8a3b7924ad7beefc0cea12d7352" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d132e75fb7746eea86aa8f7af4603cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e6c7735499b4daa9bfe9811b07dcde5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33715 } health_report { overall_health: UNKNOWN } } }
I20260927 02:37:28.085338 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:28.082301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43898 (local address 127.0.98.1:38073)
0927 02:37:28.082426 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:28.082429 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:28.082443 (+ 14us) server_negotiation.cc:407] Connection header received
0927 02:37:28.082510 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:28.082513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:28.082562 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:28.082643 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:37:28.082848 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:28.083336 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:28.084137 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:28.084339 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:28.084403 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:28.084613 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:37:28.085020 (+ 407us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:37:28.085105 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:28.085147 (+ 42us) server_negotiation.cc:299] Negotiation successful
0927 02:37:28.085194 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 02:37:28.090474 17832 raft_consensus.cc:1261] T 78e4e77bcefd45808527b93b7a7311fb P 8e6c7735499b4daa9bfe9811b07dcde5 [term 1 FOLLOWER]: Refusing update from remote peer 1d132e75fb7746eea86aa8f7af4603cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:37:28.090476 17698 raft_consensus.cc:1261] T 78e4e77bcefd45808527b93b7a7311fb P 5298b8a3b7924ad7beefc0cea12d7352 [term 1 FOLLOWER]: Refusing update from remote peer 1d132e75fb7746eea86aa8f7af4603cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:37:28.091106 17906 consensus_queue.cc:1035] T 78e4e77bcefd45808527b93b7a7311fb P 1d132e75fb7746eea86aa8f7af4603cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e6c7735499b4daa9bfe9811b07dcde5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:37:28.091270 17903 consensus_queue.cc:1035] T 78e4e77bcefd45808527b93b7a7311fb P 1d132e75fb7746eea86aa8f7af4603cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "5298b8a3b7924ad7beefc0cea12d7352" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:37:28.097383 17909 mvcc.cc:205] Tried to move back new op lower bound from 7333792350572371968 to 7333792350316728320. Current Snapshot: MvccSnapshot[applied={T|T < 7333792350572371968}]
I20260927 02:37:28.098587 17911 mvcc.cc:205] Tried to move back new op lower bound from 7333792350572371968 to 7333792350316728320. Current Snapshot: MvccSnapshot[applied={T|T < 7333792350572371968}]
I20260927 02:37:28.099025 17910 mvcc.cc:205] Tried to move back new op lower bound from 7333792350572371968 to 7333792350316728320. Current Snapshot: MvccSnapshot[applied={T|T < 7333792350572371968}]
I20260927 02:37:28.105712 17418 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:37:28.105849 17418 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:37:28.107852 17418 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e8d98b52f5db40fe8eadd23de6ed6080: Sending DeleteTablet for 3 replicas of tablet 78e4e77bcefd45808527b93b7a7311fb
I20260927 02:37:28.108652 17812 tablet_service.cc:1514] Processing DeleteTablet for tablet 78e4e77bcefd45808527b93b7a7311fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:37:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41800
I20260927 02:37:28.108678 17544 tablet_service.cc:1514] Processing DeleteTablet for tablet 78e4e77bcefd45808527b93b7a7311fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:37:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43890
I20260927 02:37:28.109097 17678 tablet_service.cc:1514] Processing DeleteTablet for tablet 78e4e77bcefd45808527b93b7a7311fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:37:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33116
I20260927 02:37:28.109431 17920 tablet_replica.cc:330] T 78e4e77bcefd45808527b93b7a7311fb P 8e6c7735499b4daa9bfe9811b07dcde5: stopping tablet replica
I20260927 02:37:28.109557 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 17481
I20260927 02:37:28.109766 17920 raft_consensus.cc:2229] T 78e4e77bcefd45808527b93b7a7311fb P 8e6c7735499b4daa9bfe9811b07dcde5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:37:28.110329 17920 raft_consensus.cc:2258] T 78e4e77bcefd45808527b93b7a7311fb P 8e6c7735499b4daa9bfe9811b07dcde5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:37:28.112422 17920 ts_tablet_manager.cc:1905] T 78e4e77bcefd45808527b93b7a7311fb P 8e6c7735499b4daa9bfe9811b07dcde5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:37:28.114508 17920 ts_tablet_manager.cc:1918] T 78e4e77bcefd45808527b93b7a7311fb P 8e6c7735499b4daa9bfe9811b07dcde5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:37:28.114621 17920 log.cc:1196] T 78e4e77bcefd45808527b93b7a7311fb P 8e6c7735499b4daa9bfe9811b07dcde5: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476410933508-392-0/minicluster-data/ts-2/wal/wals/78e4e77bcefd45808527b93b7a7311fb
W20260927 02:37:28.114665 17406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38073 (error 108)
I20260927 02:37:28.114842 17921 tablet_replica.cc:330] T 78e4e77bcefd45808527b93b7a7311fb P 5298b8a3b7924ad7beefc0cea12d7352: stopping tablet replica
I20260927 02:37:28.114912 17920 ts_tablet_manager.cc:1939] T 78e4e77bcefd45808527b93b7a7311fb P 8e6c7735499b4daa9bfe9811b07dcde5: Deleting consensus metadata
I20260927 02:37:28.115139 17921 raft_consensus.cc:2229] T 78e4e77bcefd45808527b93b7a7311fb P 5298b8a3b7924ad7beefc0cea12d7352 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:37:28.115458 17921 raft_consensus.cc:2258] T 78e4e77bcefd45808527b93b7a7311fb P 5298b8a3b7924ad7beefc0cea12d7352 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:37:28.115629 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:28.115363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38073 (local address 127.0.0.1:43904)
0927 02:37:28.115485 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:28.115552 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 02:37:28.115708 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 17615
I20260927 02:37:28.116271 17405 catalog_manager.cc:4841] TS 8e6c7735499b4daa9bfe9811b07dcde5 (127.0.98.3:33715): tablet 78e4e77bcefd45808527b93b7a7311fb (table test-table [id=2c82556f21dc4dc1bdbf7d66a934e5c4]) successfully deleted
W20260927 02:37:28.117519 17406 catalog_manager.cc:4584] TS 1d132e75fb7746eea86aa8f7af4603cf (127.0.98.1:38073): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 78e4e77bcefd45808527b93b7a7311fb: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38073: connect: Connection refused (error 111)
I20260927 02:37:28.121241 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 17749
I20260927 02:37:28.121920 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:28.121380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41557 (local address 127.0.0.1:33120)
0927 02:37:28.121804 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:28.121852 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357}
I20260927 02:37:28.125672 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 17387
2026-09-27T02:37:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2346 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T02:37:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:37:28Z Disabled control of system clock
I20260927 02:37:28.152323 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41835
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 02:37:28.255937 17926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:37:28.256222 17926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:37:28.256256 17926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:37:28.260133 17926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:37:28.260198 17926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:37:28.260212 17926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:37:28.260227 17926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:37:28.260241 17926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:37:28.264883 17926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41835
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:37:28.265988 17926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:37:28.267431 17926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:37:28.275544 17932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:28.275552 17934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:28.275573 17926 server_base.cc:1034] running on GCE node
W20260927 02:37:28.275707 17931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:28.276317 17926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:37:28.277387 17926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:37:28.278549 17926 hybrid_clock.cc:648] HybridClock initialized: now 1790476648278532 us; error 29 us; skew 500 ppm
I20260927 02:37:28.280231 17926 webserver.cc:460] Webserver started at http://127.0.98.62:34095/ using document root <none> and password file <none>
I20260927 02:37:28.280748 17926 fs_manager.cc:359] Metadata directory not provided
I20260927 02:37:28.280814 17926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:37:28.281044 17926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:37:28.282955 17926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "db6ac6dc1e274e148c93ead84ae32b79"
format_stamp: "Formatted at 2026-09-27 02:37:28 on dist-test-slave-grlk"
server_key: "19cab307114a243c13da39051a8f9264"
server_key_iv: "e2b803176b4b557f37bc76439f1459b1"
server_key_version: "encryptionkey@0"
I20260927 02:37:28.283504 17926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "db6ac6dc1e274e148c93ead84ae32b79"
format_stamp: "Formatted at 2026-09-27 02:37:28 on dist-test-slave-grlk"
server_key: "19cab307114a243c13da39051a8f9264"
server_key_iv: "e2b803176b4b557f37bc76439f1459b1"
server_key_version: "encryptionkey@0"
I20260927 02:37:28.286850 17926 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:37:28.289336 17940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:28.290518 17926 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:37:28.290659 17926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "db6ac6dc1e274e148c93ead84ae32b79"
format_stamp: "Formatted at 2026-09-27 02:37:28 on dist-test-slave-grlk"
server_key: "19cab307114a243c13da39051a8f9264"
server_key_iv: "e2b803176b4b557f37bc76439f1459b1"
server_key_version: "encryptionkey@0"
I20260927 02:37:28.290755 17926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:37:28.312057 17926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:37:28.312744 17926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:37:28.312953 17926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:37:28.319442 17926 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41835
I20260927 02:37:28.319451 17992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41835 every 8 connection(s)
I20260927 02:37:28.320317 17926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:37:28.323237 17993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:37:28.327119 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 17926
I20260927 02:37:28.327253 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:37:28.327518 392 external_mini_cluster.cc:1428] Setting key 33e0992d3b600e1639f0132f30a5b84e
I20260927 02:37:28.328821 17993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db6ac6dc1e274e148c93ead84ae32b79: Bootstrap starting.
I20260927 02:37:28.330961 17993 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db6ac6dc1e274e148c93ead84ae32b79: Neither blocks nor log segments found. Creating new log.
I20260927 02:37:28.331646 17993 log.cc:824] T 00000000000000000000000000000000 P db6ac6dc1e274e148c93ead84ae32b79: Log is configured to *not* fsync() on all Append() calls
I20260927 02:37:28.333295 17993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db6ac6dc1e274e148c93ead84ae32b79: No bootstrap required, opened a new log
I20260927 02:37:28.335719 17996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:28.328788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44226 (local address 127.0.98.62:41835)
0927 02:37:28.329382 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:28.329397 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:28.329430 (+ 33us) server_negotiation.cc:407] Connection header received
0927 02:37:28.330029 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:28.330046 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:28.330420 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:28.330679 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:28.331005 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:28.331995 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:28.332834 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:28.333333 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:28.333474 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:28.334207 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:28.334212 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:28.334224 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:37:28.334591 (+ 367us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:28.334642 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:37:28.334662 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:28.334744 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:28.334868 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 02:37:28.335096 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":137,"threads_started":1}
I20260927 02:37:28.336613 17993 raft_consensus.cc:348] T 00000000000000000000000000000000 P db6ac6dc1e274e148c93ead84ae32b79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db6ac6dc1e274e148c93ead84ae32b79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41835 } }
I20260927 02:37:28.336869 17993 raft_consensus.cc:374] T 00000000000000000000000000000000 P db6ac6dc1e274e148c93ead84ae32b79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:37:28.336925 17993 raft_consensus.cc:729] T 00000000000000000000000000000000 P db6ac6dc1e274e148c93ead84ae32b79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db6ac6dc1e274e148c93ead84ae32b79, State: Initialized, Role: FOLLOWER
I20260927 02:37:28.337376 17993 consensus_queue.cc:260] T 00000000000000000000000000000000 P db6ac6dc1e274e148c93ead84ae32b79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db6ac6dc1e274e148c93ead84ae32b79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41835 } }
I20260927 02:37:28.337523 17993 raft_consensus.cc:388] T 00000000000000000000000000000000 P db6ac6dc1e274e148c93ead84ae32b79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:37:28.337585 17993 raft_consensus.cc:482] T 00000000000000000000000000000000 P db6ac6dc1e274e148c93ead84ae32b79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:37:28.337679 17993 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db6ac6dc1e274e148c93ead84ae32b79 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:37:28.338426 17993 raft_consensus.cc:504] T 00000000000000000000000000000000 P db6ac6dc1e274e148c93ead84ae32b79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db6ac6dc1e274e148c93ead84ae32b79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41835 } }
I20260927 02:37:28.338713 17993 leader_election.cc:302] T 00000000000000000000000000000000 P db6ac6dc1e274e148c93ead84ae32b79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db6ac6dc1e274e148c93ead84ae32b79; no voters:
I20260927 02:37:28.338953 17993 leader_election.cc:288] T 00000000000000000000000000000000 P db6ac6dc1e274e148c93ead84ae32b79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:37:28.339075 17998 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db6ac6dc1e274e148c93ead84ae32b79 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:37:28.339246 17998 raft_consensus.cc:686] T 00000000000000000000000000000000 P db6ac6dc1e274e148c93ead84ae32b79 [term 1 LEADER]: Becoming Leader. State: Replica: db6ac6dc1e274e148c93ead84ae32b79, State: Running, Role: LEADER
I20260927 02:37:28.339562 17998 consensus_queue.cc:237] T 00000000000000000000000000000000 P db6ac6dc1e274e148c93ead84ae32b79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db6ac6dc1e274e148c93ead84ae32b79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41835 } }
I20260927 02:37:28.339936 17993 sys_catalog.cc:564] T 00000000000000000000000000000000 P db6ac6dc1e274e148c93ead84ae32b79 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:37:28.341164 18000 sys_catalog.cc:455] T 00000000000000000000000000000000 P db6ac6dc1e274e148c93ead84ae32b79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db6ac6dc1e274e148c93ead84ae32b79. Latest consensus state: current_term: 1 leader_uuid: "db6ac6dc1e274e148c93ead84ae32b79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db6ac6dc1e274e148c93ead84ae32b79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41835 } } }
I20260927 02:37:28.341311 18000 sys_catalog.cc:458] T 00000000000000000000000000000000 P db6ac6dc1e274e148c93ead84ae32b79 [sys.catalog]: This master's current role is: LEADER
I20260927 02:37:28.341164 17999 sys_catalog.cc:455] T 00000000000000000000000000000000 P db6ac6dc1e274e148c93ead84ae32b79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db6ac6dc1e274e148c93ead84ae32b79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db6ac6dc1e274e148c93ead84ae32b79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41835 } } }
I20260927 02:37:28.341530 17999 sys_catalog.cc:458] T 00000000000000000000000000000000 P db6ac6dc1e274e148c93ead84ae32b79 [sys.catalog]: This master's current role is: LEADER
I20260927 02:37:28.341820 18009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:37:28.344341 18009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:37:28.349395 18009 catalog_manager.cc:1347] Generated new cluster ID: ea6de417475c4eeebb220812e906e7fd
I20260927 02:37:28.349450 18009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:37:28.359668 18009 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:37:28.360754 18009 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:37:28.373615 18009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db6ac6dc1e274e148c93ead84ae32b79: Generated new TSK 0
I20260927 02:37:28.374315 18009 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:37:28.383911 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41835
--builtin_ntp_servers=127.0.98.20:45687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 02:37:28.487365 18017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:37:28.487593 18017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:37:28.487643 18017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:37:28.491516 18017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:37:28.491561 18017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:37:28.491631 18017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:37:28.496210 18017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:37:28.497135 18017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:37:28.498356 18017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:37:28.507004 18017 server_base.cc:1034] running on GCE node
W20260927 02:37:28.507032 18025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:28.506968 18022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:28.507251 18023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:28.507629 18017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:37:28.508169 18017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:37:28.509315 18017 hybrid_clock.cc:648] HybridClock initialized: now 1790476648509289 us; error 39 us; skew 500 ppm
I20260927 02:37:28.510869 18017 webserver.cc:460] Webserver started at http://127.0.98.1:41565/ using document root <none> and password file <none>
I20260927 02:37:28.511372 18017 fs_manager.cc:359] Metadata directory not provided
I20260927 02:37:28.511435 18017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:37:28.511605 18017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:37:28.513492 18017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-0/data/instance:
uuid: "1c115e56c286474782a0ee0da75adea1"
format_stamp: "Formatted at 2026-09-27 02:37:28 on dist-test-slave-grlk"
server_key: "d604b176444a19385a090d89df621d97"
server_key_iv: "defc8d27def421b33e9d9d29819eeaac"
server_key_version: "encryptionkey@0"
I20260927 02:37:28.513981 18017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1c115e56c286474782a0ee0da75adea1"
format_stamp: "Formatted at 2026-09-27 02:37:28 on dist-test-slave-grlk"
server_key: "d604b176444a19385a090d89df621d97"
server_key_iv: "defc8d27def421b33e9d9d29819eeaac"
server_key_version: "encryptionkey@0"
I20260927 02:37:28.517248 18017 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:37:28.519356 18031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:28.520406 18017 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:37:28.520504 18017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-0/wal
uuid: "1c115e56c286474782a0ee0da75adea1"
format_stamp: "Formatted at 2026-09-27 02:37:28 on dist-test-slave-grlk"
server_key: "d604b176444a19385a090d89df621d97"
server_key_iv: "defc8d27def421b33e9d9d29819eeaac"
server_key_version: "encryptionkey@0"
I20260927 02:37:28.520589 18017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:37:28.540794 18017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:37:28.541381 18017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:37:28.541527 18017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:37:28.542014 18017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:37:28.542831 18017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:37:28.542877 18017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:28.542924 18017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:37:28.542953 18017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:28.550592 18017 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37313
I20260927 02:37:28.550618 18144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37313 every 8 connection(s)
I20260927 02:37:28.551355 18017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:37:28.559067 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 18017
I20260927 02:37:28.559170 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:37:28.559464 392 external_mini_cluster.cc:1428] Setting key fc2e9b5c6e603312702327a3f54837bd
I20260927 02:37:28.559536 17996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:28.552893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52291 (local address 127.0.98.62:41835)
0927 02:37:28.553021 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:28.553024 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:28.553971 (+ 947us) server_negotiation.cc:407] Connection header received
0927 02:37:28.555258 (+ 1287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:28.555262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:28.555322 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:28.555393 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:28.556197 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:28.556727 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:28.557834 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:28.558054 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:28.558145 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:28.559088 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:28.559091 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:28.559094 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:37:28.559105 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:28.559122 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:37:28.559130 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:28.559185 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:28.559376 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 02:37:28.559426 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 02:37:28.560408 18147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:28.553263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41835 (local address 127.0.98.1:52291)
0927 02:37:28.553838 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:28.553873 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:28.555059 (+ 1186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:28.555556 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:28.555562 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:28.555639 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:37:28.556068 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:28.556079 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:28.556841 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:28.556844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:28.557718 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:28.557727 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:28.558187 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:28.558191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:28.558348 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:28.558843 (+ 495us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:37:28.558845 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:37:28.558897 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:37:28.558954 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:37:28.559219 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:37:28.559229 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:37:28.559236 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 02:37:28.559374 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 02:37:28.559633 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":101,"threads_started":1}
I20260927 02:37:28.561327 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41835
--builtin_ntp_servers=127.0.98.20:45687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 02:37:28.561517 18145 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41835
I20260927 02:37:28.561708 18145 heartbeater.cc:464] Registering TS with master...
I20260927 02:37:28.562126 18145 heartbeater.cc:511] Master 127.0.98.62:41835 requested a full tablet report, sending...
I20260927 02:37:28.563498 17956 ts_manager.cc:194] Registered new tserver with Master: 1c115e56c286474782a0ee0da75adea1 (127.0.98.1:37313)
I20260927 02:37:28.564791 17956 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:52291
W20260927 02:37:28.669704 18148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:37:28.669991 18148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:37:28.670054 18148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:37:28.674084 18148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:37:28.674131 18148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:37:28.674204 18148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:37:28.679586 18148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:37:28.680536 18148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:37:28.681824 18148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:37:28.690608 18156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:28.690714 18148 server_base.cc:1034] running on GCE node
W20260927 02:37:28.690806 18154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:28.690902 18153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:28.691203 18148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:37:28.691757 18148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:37:28.692905 18148 hybrid_clock.cc:648] HybridClock initialized: now 1790476648692893 us; error 28 us; skew 500 ppm
I20260927 02:37:28.694684 18148 webserver.cc:460] Webserver started at http://127.0.98.2:41619/ using document root <none> and password file <none>
I20260927 02:37:28.695197 18148 fs_manager.cc:359] Metadata directory not provided
I20260927 02:37:28.695268 18148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:37:28.695436 18148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:37:28.697322 18148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-1/data/instance:
uuid: "933036ce7d9846ab93dab81cd64c76d2"
format_stamp: "Formatted at 2026-09-27 02:37:28 on dist-test-slave-grlk"
server_key: "e6e260935e1d8e3780322bb3291c9df8"
server_key_iv: "a39b0ba6d0f0ba44bd7ea995c4f50d84"
server_key_version: "encryptionkey@0"
I20260927 02:37:28.697814 18148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-1/wal/instance:
uuid: "933036ce7d9846ab93dab81cd64c76d2"
format_stamp: "Formatted at 2026-09-27 02:37:28 on dist-test-slave-grlk"
server_key: "e6e260935e1d8e3780322bb3291c9df8"
server_key_iv: "a39b0ba6d0f0ba44bd7ea995c4f50d84"
server_key_version: "encryptionkey@0"
I20260927 02:37:28.701061 18148 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:37:28.703173 18162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:28.704315 18148 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:37:28.704408 18148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-1/wal
uuid: "933036ce7d9846ab93dab81cd64c76d2"
format_stamp: "Formatted at 2026-09-27 02:37:28 on dist-test-slave-grlk"
server_key: "e6e260935e1d8e3780322bb3291c9df8"
server_key_iv: "a39b0ba6d0f0ba44bd7ea995c4f50d84"
server_key_version: "encryptionkey@0"
I20260927 02:37:28.704504 18148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:37:28.735235 18148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:37:28.735913 18148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:37:28.736076 18148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:37:28.736624 18148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:37:28.737546 18148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:37:28.737592 18148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:28.737635 18148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:37:28.737663 18148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:28.745620 18148 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34833
I20260927 02:37:28.745690 18275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34833 every 8 connection(s)
I20260927 02:37:28.746440 18148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:37:28.746703 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 18148
I20260927 02:37:28.746817 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:37:28.747113 392 external_mini_cluster.cc:1428] Setting key ccc84ab97437a41daa1801990336b7d2
I20260927 02:37:28.749073 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41835
--builtin_ntp_servers=127.0.98.20:45687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 02:37:28.756320 17996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:28.748178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35475 (local address 127.0.98.62:41835)
0927 02:37:28.748286 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:28.748290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:28.749543 (+ 1253us) server_negotiation.cc:407] Connection header received
0927 02:37:28.751082 (+ 1539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:28.751084 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:28.751127 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:28.751193 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:28.752184 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:28.752794 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:28.754441 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:28.754647 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:28.754719 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:28.755888 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:28.755891 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:28.755893 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:37:28.755903 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:28.755923 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:37:28.755930 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:28.755985 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:28.756163 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 02:37:28.756212 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 02:37:28.757457 18278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:28.748659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41835 (local address 127.0.98.2:35475)
0927 02:37:28.749398 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:28.749447 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:28.750871 (+ 1424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:28.751369 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:28.751377 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:28.751472 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:37:28.752070 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:28.752086 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:28.752925 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:28.752951 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:28.754316 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:28.754327 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:28.754782 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:28.754787 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:28.754981 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:28.755591 (+ 610us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:37:28.755593 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:37:28.755648 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:37:28.755727 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:37:28.756006 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:37:28.756014 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:37:28.756020 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 02:37:28.756149 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 02:37:28.756443 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":117,"threads_started":1}
I20260927 02:37:28.758517 18276 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41835
I20260927 02:37:28.758752 18276 heartbeater.cc:464] Registering TS with master...
I20260927 02:37:28.759341 18276 heartbeater.cc:511] Master 127.0.98.62:41835 requested a full tablet report, sending...
I20260927 02:37:28.760373 17956 ts_manager.cc:194] Registered new tserver with Master: 933036ce7d9846ab93dab81cd64c76d2 (127.0.98.2:34833)
I20260927 02:37:28.761113 17956 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:35475
W20260927 02:37:28.855183 18279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:37:28.855402 18279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:37:28.855436 18279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:37:28.859295 18279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:37:28.859344 18279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:37:28.859393 18279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:37:28.864099 18279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:37:28.865118 18279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:37:28.866346 18279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:37:28.875069 18287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:28.875196 18285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:28.875283 18279 server_base.cc:1034] running on GCE node
W20260927 02:37:28.875169 18284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:28.875681 18279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:37:28.876248 18279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:37:28.877395 18279 hybrid_clock.cc:648] HybridClock initialized: now 1790476648877376 us; error 31 us; skew 500 ppm
I20260927 02:37:28.878966 18279 webserver.cc:460] Webserver started at http://127.0.98.3:46777/ using document root <none> and password file <none>
I20260927 02:37:28.879467 18279 fs_manager.cc:359] Metadata directory not provided
I20260927 02:37:28.879529 18279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:37:28.879695 18279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:37:28.881570 18279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-2/data/instance:
uuid: "fb33d4b8eebf43a5b6aef13dfd36b1a2"
format_stamp: "Formatted at 2026-09-27 02:37:28 on dist-test-slave-grlk"
server_key: "822335dc6a004b24beb66220e003c694"
server_key_iv: "7fc82d0d8ad9007c18d721cbdda9705f"
server_key_version: "encryptionkey@0"
I20260927 02:37:28.882045 18279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fb33d4b8eebf43a5b6aef13dfd36b1a2"
format_stamp: "Formatted at 2026-09-27 02:37:28 on dist-test-slave-grlk"
server_key: "822335dc6a004b24beb66220e003c694"
server_key_iv: "7fc82d0d8ad9007c18d721cbdda9705f"
server_key_version: "encryptionkey@0"
I20260927 02:37:28.885304 18279 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:37:28.887512 18293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:28.888597 18279 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:37:28.888701 18279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-2/wal
uuid: "fb33d4b8eebf43a5b6aef13dfd36b1a2"
format_stamp: "Formatted at 2026-09-27 02:37:28 on dist-test-slave-grlk"
server_key: "822335dc6a004b24beb66220e003c694"
server_key_iv: "7fc82d0d8ad9007c18d721cbdda9705f"
server_key_version: "encryptionkey@0"
I20260927 02:37:28.888787 18279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:37:28.910066 18279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:37:28.910765 18279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:37:28.910934 18279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:37:28.911499 18279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:37:28.912400 18279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:37:28.912447 18279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:28.912497 18279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:37:28.912529 18279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:28.920361 18279 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42633
I20260927 02:37:28.920411 18406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42633 every 8 connection(s)
I20260927 02:37:28.921226 18279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:37:28.924674 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 18279
I20260927 02:37:28.924763 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:37:28.925035 392 external_mini_cluster.cc:1428] Setting key a8091ff6402a610e949c480aca29ecbe
I20260927 02:37:28.930045 17996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:28.922865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57217 (local address 127.0.98.62:41835)
0927 02:37:28.923001 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:28.923005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:28.923986 (+ 981us) server_negotiation.cc:407] Connection header received
0927 02:37:28.925463 (+ 1477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:28.925466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:28.925517 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:28.925601 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:28.926704 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:28.927306 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:28.928428 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:28.928648 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:28.928724 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:28.929689 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:28.929692 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:28.929694 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:37:28.929705 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:28.929725 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:37:28.929733 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:28.929788 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:28.929891 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 02:37:28.929937 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 02:37:28.930881 18409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:28.923237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41835 (local address 127.0.98.3:57217)
0927 02:37:28.923855 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:28.923892 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:28.925246 (+ 1354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:28.925806 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:28.925815 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:28.925921 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:37:28.926561 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:28.926581 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:28.927416 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:28.927419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:28.928328 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:28.928337 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:28.928765 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:28.928768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:28.928903 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:28.929463 (+ 560us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:37:28.929465 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:37:28.929502 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:37:28.929561 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:37:28.929793 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:37:28.929799 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:37:28.929803 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:37:28.929885 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 02:37:28.930091 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":104,"threads_started":1}
I20260927 02:37:28.931845 18407 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41835
I20260927 02:37:28.932042 18407 heartbeater.cc:464] Registering TS with master...
I20260927 02:37:28.932507 18407 heartbeater.cc:511] Master 127.0.98.62:41835 requested a full tablet report, sending...
I20260927 02:37:28.933409 17956 ts_manager.cc:194] Registered new tserver with Master: fb33d4b8eebf43a5b6aef13dfd36b1a2 (127.0.98.3:42633)
I20260927 02:37:28.933934 17956 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:57217
I20260927 02:37:28.938373 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:37:28.943563 17996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:28.940121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44228 (local address 127.0.98.62:41835)
0927 02:37:28.940247 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:28.940267 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:28.940414 (+ 147us) server_negotiation.cc:407] Connection header received
0927 02:37:28.940498 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:28.940500 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:28.940538 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:28.940599 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:28.940789 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:28.941369 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:28.942177 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:28.942381 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:28.942455 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:28.943167 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:28.943170 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:28.943172 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:37:28.943183 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:28.943202 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:37:28.943210 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:28.943273 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:28.943416 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 02:37:28.943462 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 02:37:28.946041 17956 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44228:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:37:28.948902 17956 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:37:28.962785 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:28.956398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37313 (local address 127.0.0.1:48544)
0927 02:37:28.957186 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:28.957253 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:28.957537 (+ 284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:28.958265 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:28.958271 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:28.958294 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:37:28.958438 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:28.958446 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:28.959945 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:28.959949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:28.961473 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:28.961481 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:28.962234 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:28.962237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:28.962340 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:28.962378 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 02:37:28.962496 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 02:37:28.962622 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"mutex_wait_us":255,"thread_start_us":86,"threads_started":1}
I20260927 02:37:28.962787 18420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:28.956545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48544 (local address 127.0.98.1:37313)
0927 02:37:28.956868 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:28.956879 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:28.957765 (+ 886us) server_negotiation.cc:407] Connection header received
0927 02:37:28.957856 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:28.957862 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:28.958089 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:28.958265 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:37:28.958571 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:28.959806 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:28.961564 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:28.962114 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:28.962212 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:28.962434 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:28.962526 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 02:37:28.962627 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":125,"threads_started":1}
I20260927 02:37:28.962975 18418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:28.957167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42633 (local address 127.0.0.1:48644)
0927 02:37:28.958279 (+ 1112us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:28.958292 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:28.958371 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:28.958973 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:28.958975 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:28.958989 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:37:28.959051 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:28.959055 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:28.960067 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:28.960071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:28.961131 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:28.961142 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:28.962580 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:28.962584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:28.962667 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:28.962689 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 02:37:28.962752 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 02:37:28.962823 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1045,"mutex_wait_us":8,"thread_start_us":89,"threads_started":1}
I20260927 02:37:28.963165 18421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:28.956462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34833 (local address 127.0.0.1:39190)
0927 02:37:28.957909 (+ 1447us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:28.957923 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:28.958010 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:28.958685 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:28.958688 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:28.958704 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:37:28.958802 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:28.958807 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:28.960124 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:28.960127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:28.961369 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:28.961380 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:28.962820 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:28.962823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:28.962904 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:28.962923 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 02:37:28.962992 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 02:37:28.963061 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1367,"thread_start_us":95,"threads_started":1}
I20260927 02:37:28.963500 18423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:28.957553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39190 (local address 127.0.98.2:34833)
0927 02:37:28.957985 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:28.957996 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:28.958018 (+ 22us) server_negotiation.cc:407] Connection header received
0927 02:37:28.958330 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:28.958337 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:28.958548 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:28.958699 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:37:28.958910 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:28.960011 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:28.961498 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:28.962497 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:28.962593 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:28.962752 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:28.963249 (+ 497us) server_negotiation.cc:299] Negotiation successful
0927 02:37:28.963343 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":138,"threads_started":1}
I20260927 02:37:28.963716 18422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:28.957385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48644 (local address 127.0.98.3:42633)
0927 02:37:28.958010 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:28.958021 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:28.958442 (+ 421us) server_negotiation.cc:407] Connection header received
0927 02:37:28.958524 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:28.958528 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:28.958701 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:28.958841 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:37:28.959155 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:28.959939 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:28.961245 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:28.962204 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:28.962339 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:28.962515 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:28.963072 (+ 557us) server_negotiation.cc:299] Negotiation successful
0927 02:37:28.963562 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":103,"threads_started":1}
I20260927 02:37:28.966066 18079 tablet_service.cc:1467] Processing CreateTablet for tablet f4e9ba1dde5b4d689402ee2b4da1d892 (DEFAULT_TABLE table=test-table [id=f04a40764c3143dabf061473550b06cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:37:28.966187 18341 tablet_service.cc:1467] Processing CreateTablet for tablet f4e9ba1dde5b4d689402ee2b4da1d892 (DEFAULT_TABLE table=test-table [id=f04a40764c3143dabf061473550b06cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:37:28.966564 18210 tablet_service.cc:1467] Processing CreateTablet for tablet f4e9ba1dde5b4d689402ee2b4da1d892 (DEFAULT_TABLE table=test-table [id=f04a40764c3143dabf061473550b06cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:37:28.966995 18079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4e9ba1dde5b4d689402ee2b4da1d892. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:37:28.967147 18341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4e9ba1dde5b4d689402ee2b4da1d892. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:37:28.967393 18210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4e9ba1dde5b4d689402ee2b4da1d892. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:37:28.973402 18424 tablet_bootstrap.cc:492] T f4e9ba1dde5b4d689402ee2b4da1d892 P 1c115e56c286474782a0ee0da75adea1: Bootstrap starting.
I20260927 02:37:28.974072 18426 tablet_bootstrap.cc:492] T f4e9ba1dde5b4d689402ee2b4da1d892 P fb33d4b8eebf43a5b6aef13dfd36b1a2: Bootstrap starting.
I20260927 02:37:28.974073 18425 tablet_bootstrap.cc:492] T f4e9ba1dde5b4d689402ee2b4da1d892 P 933036ce7d9846ab93dab81cd64c76d2: Bootstrap starting.
I20260927 02:37:28.975360 18424 tablet_bootstrap.cc:654] T f4e9ba1dde5b4d689402ee2b4da1d892 P 1c115e56c286474782a0ee0da75adea1: Neither blocks nor log segments found. Creating new log.
I20260927 02:37:28.976012 18424 log.cc:824] T f4e9ba1dde5b4d689402ee2b4da1d892 P 1c115e56c286474782a0ee0da75adea1: Log is configured to *not* fsync() on all Append() calls
I20260927 02:37:28.976059 18425 tablet_bootstrap.cc:654] T f4e9ba1dde5b4d689402ee2b4da1d892 P 933036ce7d9846ab93dab81cd64c76d2: Neither blocks nor log segments found. Creating new log.
I20260927 02:37:28.976059 18426 tablet_bootstrap.cc:654] T f4e9ba1dde5b4d689402ee2b4da1d892 P fb33d4b8eebf43a5b6aef13dfd36b1a2: Neither blocks nor log segments found. Creating new log.
I20260927 02:37:28.976815 18426 log.cc:824] T f4e9ba1dde5b4d689402ee2b4da1d892 P fb33d4b8eebf43a5b6aef13dfd36b1a2: Log is configured to *not* fsync() on all Append() calls
I20260927 02:37:28.976815 18425 log.cc:824] T f4e9ba1dde5b4d689402ee2b4da1d892 P 933036ce7d9846ab93dab81cd64c76d2: Log is configured to *not* fsync() on all Append() calls
I20260927 02:37:28.977763 18424 tablet_bootstrap.cc:492] T f4e9ba1dde5b4d689402ee2b4da1d892 P 1c115e56c286474782a0ee0da75adea1: No bootstrap required, opened a new log
I20260927 02:37:28.977936 18424 ts_tablet_manager.cc:1397] T f4e9ba1dde5b4d689402ee2b4da1d892 P 1c115e56c286474782a0ee0da75adea1: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 02:37:28.978574 18425 tablet_bootstrap.cc:492] T f4e9ba1dde5b4d689402ee2b4da1d892 P 933036ce7d9846ab93dab81cd64c76d2: No bootstrap required, opened a new log
I20260927 02:37:28.978570 18426 tablet_bootstrap.cc:492] T f4e9ba1dde5b4d689402ee2b4da1d892 P fb33d4b8eebf43a5b6aef13dfd36b1a2: No bootstrap required, opened a new log
I20260927 02:37:28.978739 18426 ts_tablet_manager.cc:1397] T f4e9ba1dde5b4d689402ee2b4da1d892 P fb33d4b8eebf43a5b6aef13dfd36b1a2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 02:37:28.978739 18425 ts_tablet_manager.cc:1397] T f4e9ba1dde5b4d689402ee2b4da1d892 P 933036ce7d9846ab93dab81cd64c76d2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 02:37:28.981415 18424 raft_consensus.cc:348] T f4e9ba1dde5b4d689402ee2b4da1d892 P 1c115e56c286474782a0ee0da75adea1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c115e56c286474782a0ee0da75adea1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37313 } } peers { permanent_uuid: "933036ce7d9846ab93dab81cd64c76d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34833 } } peers { permanent_uuid: "fb33d4b8eebf43a5b6aef13dfd36b1a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42633 } }
I20260927 02:37:28.981611 18424 raft_consensus.cc:374] T f4e9ba1dde5b4d689402ee2b4da1d892 P 1c115e56c286474782a0ee0da75adea1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:37:28.981684 18424 raft_consensus.cc:729] T f4e9ba1dde5b4d689402ee2b4da1d892 P 1c115e56c286474782a0ee0da75adea1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c115e56c286474782a0ee0da75adea1, State: Initialized, Role: FOLLOWER
I20260927 02:37:28.982079 18424 consensus_queue.cc:260] T f4e9ba1dde5b4d689402ee2b4da1d892 P 1c115e56c286474782a0ee0da75adea1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c115e56c286474782a0ee0da75adea1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37313 } } peers { permanent_uuid: "933036ce7d9846ab93dab81cd64c76d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34833 } } peers { permanent_uuid: "fb33d4b8eebf43a5b6aef13dfd36b1a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42633 } }
I20260927 02:37:28.982625 18145 heartbeater.cc:503] Master 127.0.98.62:41835 was elected leader, sending a full tablet report...
I20260927 02:37:28.982636 18426 raft_consensus.cc:348] T f4e9ba1dde5b4d689402ee2b4da1d892 P fb33d4b8eebf43a5b6aef13dfd36b1a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c115e56c286474782a0ee0da75adea1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37313 } } peers { permanent_uuid: "933036ce7d9846ab93dab81cd64c76d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34833 } } peers { permanent_uuid: "fb33d4b8eebf43a5b6aef13dfd36b1a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42633 } }
I20260927 02:37:28.982636 18425 raft_consensus.cc:348] T f4e9ba1dde5b4d689402ee2b4da1d892 P 933036ce7d9846ab93dab81cd64c76d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c115e56c286474782a0ee0da75adea1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37313 } } peers { permanent_uuid: "933036ce7d9846ab93dab81cd64c76d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34833 } } peers { permanent_uuid: "fb33d4b8eebf43a5b6aef13dfd36b1a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42633 } }
I20260927 02:37:28.982852 18425 raft_consensus.cc:374] T f4e9ba1dde5b4d689402ee2b4da1d892 P 933036ce7d9846ab93dab81cd64c76d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:37:28.982853 18426 raft_consensus.cc:374] T f4e9ba1dde5b4d689402ee2b4da1d892 P fb33d4b8eebf43a5b6aef13dfd36b1a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:37:28.982898 18426 raft_consensus.cc:729] T f4e9ba1dde5b4d689402ee2b4da1d892 P fb33d4b8eebf43a5b6aef13dfd36b1a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb33d4b8eebf43a5b6aef13dfd36b1a2, State: Initialized, Role: FOLLOWER
I20260927 02:37:28.982898 18425 raft_consensus.cc:729] T f4e9ba1dde5b4d689402ee2b4da1d892 P 933036ce7d9846ab93dab81cd64c76d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 933036ce7d9846ab93dab81cd64c76d2, State: Initialized, Role: FOLLOWER
I20260927 02:37:28.982976 18424 ts_tablet_manager.cc:1428] T f4e9ba1dde5b4d689402ee2b4da1d892 P 1c115e56c286474782a0ee0da75adea1: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 02:37:28.983323 18425 consensus_queue.cc:260] T f4e9ba1dde5b4d689402ee2b4da1d892 P 933036ce7d9846ab93dab81cd64c76d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c115e56c286474782a0ee0da75adea1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37313 } } peers { permanent_uuid: "933036ce7d9846ab93dab81cd64c76d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34833 } } peers { permanent_uuid: "fb33d4b8eebf43a5b6aef13dfd36b1a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42633 } }
I20260927 02:37:28.983323 18426 consensus_queue.cc:260] T f4e9ba1dde5b4d689402ee2b4da1d892 P fb33d4b8eebf43a5b6aef13dfd36b1a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c115e56c286474782a0ee0da75adea1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37313 } } peers { permanent_uuid: "933036ce7d9846ab93dab81cd64c76d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34833 } } peers { permanent_uuid: "fb33d4b8eebf43a5b6aef13dfd36b1a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42633 } }
I20260927 02:37:28.983414 18430 raft_consensus.cc:482] T f4e9ba1dde5b4d689402ee2b4da1d892 P 1c115e56c286474782a0ee0da75adea1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:37:28.983693 18430 raft_consensus.cc:504] T f4e9ba1dde5b4d689402ee2b4da1d892 P 1c115e56c286474782a0ee0da75adea1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c115e56c286474782a0ee0da75adea1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37313 } } peers { permanent_uuid: "933036ce7d9846ab93dab81cd64c76d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34833 } } peers { permanent_uuid: "fb33d4b8eebf43a5b6aef13dfd36b1a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42633 } }
I20260927 02:37:28.983904 18407 heartbeater.cc:503] Master 127.0.98.62:41835 was elected leader, sending a full tablet report...
I20260927 02:37:28.984189 18426 ts_tablet_manager.cc:1428] T f4e9ba1dde5b4d689402ee2b4da1d892 P fb33d4b8eebf43a5b6aef13dfd36b1a2: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 02:37:28.986521 18425 ts_tablet_manager.cc:1428] T f4e9ba1dde5b4d689402ee2b4da1d892 P 933036ce7d9846ab93dab81cd64c76d2: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260927 02:37:28.986661 18430 leader_election.cc:288] T f4e9ba1dde5b4d689402ee2b4da1d892 P 1c115e56c286474782a0ee0da75adea1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 933036ce7d9846ab93dab81cd64c76d2 (127.0.98.2:34833), fb33d4b8eebf43a5b6aef13dfd36b1a2 (127.0.98.3:42633)
I20260927 02:37:28.986868 18276 heartbeater.cc:503] Master 127.0.98.62:41835 was elected leader, sending a full tablet report...
I20260927 02:37:28.990408 18147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:28.984961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34833 (local address 127.0.98.1:38131)
0927 02:37:28.985309 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:28.985323 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:28.985395 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:28.985834 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:28.985838 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:28.985860 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:37:28.985968 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:28.985976 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:28.988155 (+ 2179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:28.988159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:28.989275 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:28.989284 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:28.990077 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:28.990080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:28.990164 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:28.990184 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 02:37:28.990242 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 02:37:28.990300 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":9}
I20260927 02:37:28.990437 18423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:28.985527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38131 (local address 127.0.98.2:34833)
0927 02:37:28.985627 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:28.985632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:28.985644 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:37:28.985679 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:28.985682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:28.985722 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:28.985796 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:37:28.986096 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:28.988056 (+ 1960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:28.989404 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:28.989975 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:28.990074 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:28.990129 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:28.990272 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 02:37:28.990333 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 02:37:28.990967 18230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4e9ba1dde5b4d689402ee2b4da1d892" candidate_uuid: "1c115e56c286474782a0ee0da75adea1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "933036ce7d9846ab93dab81cd64c76d2" is_pre_election: true
I20260927 02:37:28.991168 18230 raft_consensus.cc:2454] T f4e9ba1dde5b4d689402ee2b4da1d892 P 933036ce7d9846ab93dab81cd64c76d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c115e56c286474782a0ee0da75adea1 in term 0.
I20260927 02:37:28.991591 18032 leader_election.cc:302] T f4e9ba1dde5b4d689402ee2b4da1d892 P 1c115e56c286474782a0ee0da75adea1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c115e56c286474782a0ee0da75adea1, 933036ce7d9846ab93dab81cd64c76d2; no voters:
I20260927 02:37:28.991681 18422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:28.986285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32997 (local address 127.0.98.3:42633)
0927 02:37:28.986629 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:28.986634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:28.988334 (+ 1700us) server_negotiation.cc:407] Connection header received
0927 02:37:28.988439 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:28.988442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:28.988482 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:28.988556 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:37:28.988809 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:28.989335 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:28.990663 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:28.991165 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:28.991248 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:28.991298 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:28.991517 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 02:37:28.991580 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269}
I20260927 02:37:28.991712 18434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:28.986343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42633 (local address 127.0.98.1:32997)
0927 02:37:28.988254 (+ 1911us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:28.988266 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:28.988351 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:28.988578 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:28.988581 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:28.988598 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:37:28.988699 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:28.988704 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:28.989461 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:28.989464 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:28.990546 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:28.990556 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:28.991304 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:28.991307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:28.991398 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:28.991420 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 02:37:28.991490 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 02:37:28.991549 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1850,"thread_start_us":119,"threads_started":1}
I20260927 02:37:28.991957 18431 raft_consensus.cc:2781] T f4e9ba1dde5b4d689402ee2b4da1d892 P 1c115e56c286474782a0ee0da75adea1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:37:28.992029 18431 raft_consensus.cc:482] T f4e9ba1dde5b4d689402ee2b4da1d892 P 1c115e56c286474782a0ee0da75adea1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:37:28.992079 18431 raft_consensus.cc:3037] T f4e9ba1dde5b4d689402ee2b4da1d892 P 1c115e56c286474782a0ee0da75adea1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:37:28.992251 18361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4e9ba1dde5b4d689402ee2b4da1d892" candidate_uuid: "1c115e56c286474782a0ee0da75adea1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb33d4b8eebf43a5b6aef13dfd36b1a2" is_pre_election: true
I20260927 02:37:28.992451 18361 raft_consensus.cc:2454] T f4e9ba1dde5b4d689402ee2b4da1d892 P fb33d4b8eebf43a5b6aef13dfd36b1a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c115e56c286474782a0ee0da75adea1 in term 0.
I20260927 02:37:28.993134 18431 raft_consensus.cc:504] T f4e9ba1dde5b4d689402ee2b4da1d892 P 1c115e56c286474782a0ee0da75adea1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c115e56c286474782a0ee0da75adea1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37313 } } peers { permanent_uuid: "933036ce7d9846ab93dab81cd64c76d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34833 } } peers { permanent_uuid: "fb33d4b8eebf43a5b6aef13dfd36b1a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42633 } }
I20260927 02:37:28.993415 18431 leader_election.cc:288] T f4e9ba1dde5b4d689402ee2b4da1d892 P 1c115e56c286474782a0ee0da75adea1 [CANDIDATE]: Term 1 election: Requested vote from peers 933036ce7d9846ab93dab81cd64c76d2 (127.0.98.2:34833), fb33d4b8eebf43a5b6aef13dfd36b1a2 (127.0.98.3:42633)
I20260927 02:37:28.993742 18361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4e9ba1dde5b4d689402ee2b4da1d892" candidate_uuid: "1c115e56c286474782a0ee0da75adea1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb33d4b8eebf43a5b6aef13dfd36b1a2"
I20260927 02:37:28.993826 18230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4e9ba1dde5b4d689402ee2b4da1d892" candidate_uuid: "1c115e56c286474782a0ee0da75adea1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "933036ce7d9846ab93dab81cd64c76d2"
I20260927 02:37:28.993871 18361 raft_consensus.cc:3037] T f4e9ba1dde5b4d689402ee2b4da1d892 P fb33d4b8eebf43a5b6aef13dfd36b1a2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:37:28.993921 18230 raft_consensus.cc:3037] T f4e9ba1dde5b4d689402ee2b4da1d892 P 933036ce7d9846ab93dab81cd64c76d2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:37:28.994975 18361 raft_consensus.cc:2454] T f4e9ba1dde5b4d689402ee2b4da1d892 P fb33d4b8eebf43a5b6aef13dfd36b1a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c115e56c286474782a0ee0da75adea1 in term 1.
I20260927 02:37:28.994975 18230 raft_consensus.cc:2454] T f4e9ba1dde5b4d689402ee2b4da1d892 P 933036ce7d9846ab93dab81cd64c76d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c115e56c286474782a0ee0da75adea1 in term 1.
I20260927 02:37:28.995276 18032 leader_election.cc:302] T f4e9ba1dde5b4d689402ee2b4da1d892 P 1c115e56c286474782a0ee0da75adea1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c115e56c286474782a0ee0da75adea1, 933036ce7d9846ab93dab81cd64c76d2; no voters:
I20260927 02:37:28.995409 18431 raft_consensus.cc:2781] T f4e9ba1dde5b4d689402ee2b4da1d892 P 1c115e56c286474782a0ee0da75adea1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:37:28.995493 18431 raft_consensus.cc:686] T f4e9ba1dde5b4d689402ee2b4da1d892 P 1c115e56c286474782a0ee0da75adea1 [term 1 LEADER]: Becoming Leader. State: Replica: 1c115e56c286474782a0ee0da75adea1, State: Running, Role: LEADER
I20260927 02:37:28.995767 18431 consensus_queue.cc:237] T f4e9ba1dde5b4d689402ee2b4da1d892 P 1c115e56c286474782a0ee0da75adea1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c115e56c286474782a0ee0da75adea1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37313 } } peers { permanent_uuid: "933036ce7d9846ab93dab81cd64c76d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34833 } } peers { permanent_uuid: "fb33d4b8eebf43a5b6aef13dfd36b1a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42633 } }
W20260927 02:37:28.997380 18277 tablet.cc:2369] T f4e9ba1dde5b4d689402ee2b4da1d892 P 933036ce7d9846ab93dab81cd64c76d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:37:28.998454 17957 catalog_manager.cc:5495] T f4e9ba1dde5b4d689402ee2b4da1d892 P 1c115e56c286474782a0ee0da75adea1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c115e56c286474782a0ee0da75adea1 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "1c115e56c286474782a0ee0da75adea1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c115e56c286474782a0ee0da75adea1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "933036ce7d9846ab93dab81cd64c76d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb33d4b8eebf43a5b6aef13dfd36b1a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42633 } health_report { overall_health: UNKNOWN } } }
I20260927 02:37:29.013423 18420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:29.010104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48548 (local address 127.0.98.1:37313)
0927 02:37:29.010239 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:29.010244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:29.010258 (+ 14us) server_negotiation.cc:407] Connection header received
0927 02:37:29.010344 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:29.010346 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:29.010402 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:29.010491 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:37:29.010716 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:29.011220 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:29.012037 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:29.012254 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:29.012340 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:29.012616 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:37:29.013072 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:37:29.013177 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:29.013229 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 02:37:29.013284 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 02:37:29.018930 18361 raft_consensus.cc:1261] T f4e9ba1dde5b4d689402ee2b4da1d892 P fb33d4b8eebf43a5b6aef13dfd36b1a2 [term 1 FOLLOWER]: Refusing update from remote peer 1c115e56c286474782a0ee0da75adea1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:37:29.018930 18230 raft_consensus.cc:1261] T f4e9ba1dde5b4d689402ee2b4da1d892 P 933036ce7d9846ab93dab81cd64c76d2 [term 1 FOLLOWER]: Refusing update from remote peer 1c115e56c286474782a0ee0da75adea1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:37:29.019594 18431 consensus_queue.cc:1035] T f4e9ba1dde5b4d689402ee2b4da1d892 P 1c115e56c286474782a0ee0da75adea1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb33d4b8eebf43a5b6aef13dfd36b1a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:37:29.019773 18430 consensus_queue.cc:1035] T f4e9ba1dde5b4d689402ee2b4da1d892 P 1c115e56c286474782a0ee0da75adea1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "933036ce7d9846ab93dab81cd64c76d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:37:29.025709 18437 mvcc.cc:205] Tried to move back new op lower bound from 7333792354374574080 to 7333792354288582656. Current Snapshot: MvccSnapshot[applied={T|T < 7333792354374574080}]
I20260927 02:37:29.027675 18438 mvcc.cc:205] Tried to move back new op lower bound from 7333792354374574080 to 7333792354288582656. Current Snapshot: MvccSnapshot[applied={T|T < 7333792354374574080}]
I20260927 02:37:29.034796 17953 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:37:29.034974 17953 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:37:29.037403 17953 catalog_manager.cc:5782] T 00000000000000000000000000000000 P db6ac6dc1e274e148c93ead84ae32b79: Sending DeleteTablet for 3 replicas of tablet f4e9ba1dde5b4d689402ee2b4da1d892
I20260927 02:37:29.038080 18079 tablet_service.cc:1514] Processing DeleteTablet for tablet f4e9ba1dde5b4d689402ee2b4da1d892 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:37:29 UTC) from {username='slave'} at 127.0.0.1:48544
I20260927 02:37:29.038183 18210 tablet_service.cc:1514] Processing DeleteTablet for tablet f4e9ba1dde5b4d689402ee2b4da1d892 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:37:29 UTC) from {username='slave'} at 127.0.0.1:39190
I20260927 02:37:29.038617 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 18017
I20260927 02:37:29.038727 18448 tablet_replica.cc:330] T f4e9ba1dde5b4d689402ee2b4da1d892 P 933036ce7d9846ab93dab81cd64c76d2: stopping tablet replica
I20260927 02:37:29.038805 18341 tablet_service.cc:1514] Processing DeleteTablet for tablet f4e9ba1dde5b4d689402ee2b4da1d892 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:37:29 UTC) from {username='slave'} at 127.0.0.1:48644
I20260927 02:37:29.039041 18448 raft_consensus.cc:2229] T f4e9ba1dde5b4d689402ee2b4da1d892 P 933036ce7d9846ab93dab81cd64c76d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:37:29.039302 18448 raft_consensus.cc:2258] T f4e9ba1dde5b4d689402ee2b4da1d892 P 933036ce7d9846ab93dab81cd64c76d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:37:29.040899 18448 ts_tablet_manager.cc:1905] T f4e9ba1dde5b4d689402ee2b4da1d892 P 933036ce7d9846ab93dab81cd64c76d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:37:29.041141 18449 tablet_replica.cc:330] T f4e9ba1dde5b4d689402ee2b4da1d892 P fb33d4b8eebf43a5b6aef13dfd36b1a2: stopping tablet replica
I20260927 02:37:29.041445 18449 raft_consensus.cc:2229] T f4e9ba1dde5b4d689402ee2b4da1d892 P fb33d4b8eebf43a5b6aef13dfd36b1a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:37:29.041699 18449 raft_consensus.cc:2258] T f4e9ba1dde5b4d689402ee2b4da1d892 P fb33d4b8eebf43a5b6aef13dfd36b1a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:37:29.042737 18449 ts_tablet_manager.cc:1905] T f4e9ba1dde5b4d689402ee2b4da1d892 P fb33d4b8eebf43a5b6aef13dfd36b1a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:37:29.043962 18448 ts_tablet_manager.cc:1918] T f4e9ba1dde5b4d689402ee2b4da1d892 P 933036ce7d9846ab93dab81cd64c76d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:37:29.044054 18448 log.cc:1196] T f4e9ba1dde5b4d689402ee2b4da1d892 P 933036ce7d9846ab93dab81cd64c76d2: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-1/wal/wals/f4e9ba1dde5b4d689402ee2b4da1d892
I20260927 02:37:29.044386 18448 ts_tablet_manager.cc:1939] T f4e9ba1dde5b4d689402ee2b4da1d892 P 933036ce7d9846ab93dab81cd64c76d2: Deleting consensus metadata
I20260927 02:37:29.044922 18449 ts_tablet_manager.cc:1918] T f4e9ba1dde5b4d689402ee2b4da1d892 P fb33d4b8eebf43a5b6aef13dfd36b1a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:37:29.045037 18449 log.cc:1196] T f4e9ba1dde5b4d689402ee2b4da1d892 P fb33d4b8eebf43a5b6aef13dfd36b1a2: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476410933508-392-0/minicluster-data/ts-2/wal/wals/f4e9ba1dde5b4d689402ee2b4da1d892
I20260927 02:37:29.045351 18449 ts_tablet_manager.cc:1939] T f4e9ba1dde5b4d689402ee2b4da1d892 P fb33d4b8eebf43a5b6aef13dfd36b1a2: Deleting consensus metadata
I20260927 02:37:29.045555 17941 catalog_manager.cc:4841] TS 933036ce7d9846ab93dab81cd64c76d2 (127.0.98.2:34833): tablet f4e9ba1dde5b4d689402ee2b4da1d892 (table test-table [id=f04a40764c3143dabf061473550b06cb]) successfully deleted
W20260927 02:37:29.045671 17943 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37313 (error 108)
I20260927 02:37:29.046317 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 18148
I20260927 02:37:29.046387 17941 catalog_manager.cc:4841] TS fb33d4b8eebf43a5b6aef13dfd36b1a2 (127.0.98.3:42633): tablet f4e9ba1dde5b4d689402ee2b4da1d892 (table test-table [id=f04a40764c3143dabf061473550b06cb]) successfully deleted
I20260927 02:37:29.046608 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:29.046322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37313 (local address 127.0.0.1:48562)
0927 02:37:29.046476 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:29.046538 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260927 02:37:29.046744 17943 catalog_manager.cc:4584] TS 1c115e56c286474782a0ee0da75adea1 (127.0.98.1:37313): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f4e9ba1dde5b4d689402ee2b4da1d892: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37313: connect: Connection refused (error 111)
I20260927 02:37:29.051265 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 18279
I20260927 02:37:29.055601 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 17926
2026-09-27T02:37:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (928 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:37:29 dist-test-slave-grlk krb5kdc[18453](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:37:29 dist-test-slave-grlk krb5kdc[18453](info): set up 2 sockets
Sep 27 02:37:29 dist-test-slave-grlk krb5kdc[18453](info): commencing operation
krb5kdc: starting...
W20260927 02:37:29.992502 392 mini_kdc.cc:121] Time spent starting KDC: real 0.918s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/krb5kdc/test-user.keytab.
Sep 27 02:37:30 dist-test-slave-grlk krb5kdc[18453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:37:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:37:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:37:30.105592 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36067
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36067
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:37:30.208768 18466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:37:30.209040 18466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:37:30.209082 18466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:37:30.212968 18466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:37:30.213020 18466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:37:30.213038 18466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:37:30.213052 18466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:37:30.213073 18466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:37:30.218082 18466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:37:30.218945 18466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:37:30.219854 18466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:37:30.227543 18472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:30.227536 18474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:30.227538 18471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:30.227576 18466 server_base.cc:1034] running on GCE node
I20260927 02:37:30.228395 18466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:37:30.229408 18466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:37:30.230554 18466 hybrid_clock.cc:648] HybridClock initialized: now 1790476650230530 us; error 37 us; skew 500 ppm
Sep 27 02:37:30 dist-test-slave-grlk krb5kdc[18453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:37:30.232990 18466 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:37:30.233789 18466 webserver.cc:460] Webserver started at http://127.0.98.62:33997/ using document root <none> and password file <none>
I20260927 02:37:30.234272 18466 fs_manager.cc:359] Metadata directory not provided
I20260927 02:37:30.234333 18466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:37:30.234516 18466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:37:30.236361 18466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "0ce9f400aad740d7888d440be4982296"
format_stamp: "Formatted at 2026-09-27 02:37:30 on dist-test-slave-grlk"
server_key: "42a480a533df39287b07bc533d22985c"
server_key_iv: "0124077566bbc7394351becaf0e557e3"
server_key_version: "encryptionkey@0"
I20260927 02:37:30.236855 18466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "0ce9f400aad740d7888d440be4982296"
format_stamp: "Formatted at 2026-09-27 02:37:30 on dist-test-slave-grlk"
server_key: "42a480a533df39287b07bc533d22985c"
server_key_iv: "0124077566bbc7394351becaf0e557e3"
server_key_version: "encryptionkey@0"
I20260927 02:37:30.240134 18466 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 02:37:30.242347 18481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:30.243503 18466 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:37:30.243609 18466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "0ce9f400aad740d7888d440be4982296"
format_stamp: "Formatted at 2026-09-27 02:37:30 on dist-test-slave-grlk"
server_key: "42a480a533df39287b07bc533d22985c"
server_key_iv: "0124077566bbc7394351becaf0e557e3"
server_key_version: "encryptionkey@0"
I20260927 02:37:30.243703 18466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:37:30.266606 18466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:37:30.269222 18466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:37:30.269387 18466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:37:30.275905 18466 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36067
I20260927 02:37:30.275920 18533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36067 every 8 connection(s)
I20260927 02:37:30.276814 18466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:37:30.279326 18534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:37:30.280462 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 18466
I20260927 02:37:30.280570 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:37:30.280828 392 external_mini_cluster.cc:1428] Setting key 688eaa8f19f51302512d96791708b276
I20260927 02:37:30.285494 18534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ce9f400aad740d7888d440be4982296: Bootstrap starting.
I20260927 02:37:30.287957 18534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ce9f400aad740d7888d440be4982296: Neither blocks nor log segments found. Creating new log.
I20260927 02:37:30.288693 18534 log.cc:824] T 00000000000000000000000000000000 P 0ce9f400aad740d7888d440be4982296: Log is configured to *not* fsync() on all Append() calls
Sep 27 02:37:30 dist-test-slave-grlk krb5kdc[18453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:37:30.290702 18534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ce9f400aad740d7888d440be4982296: No bootstrap required, opened a new log
I20260927 02:37:30.294382 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:30.282109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43302 (local address 127.0.98.62:36067)
0927 02:37:30.282513 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:30.282531 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:30.282571 (+ 40us) server_negotiation.cc:407] Connection header received
0927 02:37:30.283362 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:30.283384 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:30.283796 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:30.284141 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:30.284768 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:30.286161 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:30.286965 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:30.287616 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:30.287760 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:30.290215 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:30.290219 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:30.290232 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:37:30.290266 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:30.291792 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:30.292228 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:30.292232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:30.292236 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:30.292340 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:30.292594 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:30.292596 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:30.292597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:30.292977 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:30.293125 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:30.293382 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 02:37:30.293641 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":110,"threads_started":1}
I20260927 02:37:30.294420 18534 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ce9f400aad740d7888d440be4982296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce9f400aad740d7888d440be4982296" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36067 } }
I20260927 02:37:30.294664 18534 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ce9f400aad740d7888d440be4982296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:37:30.294721 18534 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ce9f400aad740d7888d440be4982296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ce9f400aad740d7888d440be4982296, State: Initialized, Role: FOLLOWER
I20260927 02:37:30.295114 18534 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ce9f400aad740d7888d440be4982296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce9f400aad740d7888d440be4982296" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36067 } }
I20260927 02:37:30.295228 18534 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ce9f400aad740d7888d440be4982296 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:37:30.295286 18534 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ce9f400aad740d7888d440be4982296 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:37:30.295382 18534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ce9f400aad740d7888d440be4982296 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:37:30.296190 18534 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ce9f400aad740d7888d440be4982296 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce9f400aad740d7888d440be4982296" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36067 } }
I20260927 02:37:30.296543 18534 leader_election.cc:302] T 00000000000000000000000000000000 P 0ce9f400aad740d7888d440be4982296 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ce9f400aad740d7888d440be4982296; no voters:
I20260927 02:37:30.296787 18534 leader_election.cc:288] T 00000000000000000000000000000000 P 0ce9f400aad740d7888d440be4982296 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:37:30.296890 18539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ce9f400aad740d7888d440be4982296 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:37:30.297104 18539 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ce9f400aad740d7888d440be4982296 [term 1 LEADER]: Becoming Leader. State: Replica: 0ce9f400aad740d7888d440be4982296, State: Running, Role: LEADER
I20260927 02:37:30.297407 18539 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ce9f400aad740d7888d440be4982296 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce9f400aad740d7888d440be4982296" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36067 } }
I20260927 02:37:30.297866 18534 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ce9f400aad740d7888d440be4982296 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:37:30.298796 18540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ce9f400aad740d7888d440be4982296 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ce9f400aad740d7888d440be4982296" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce9f400aad740d7888d440be4982296" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36067 } } }
I20260927 02:37:30.298938 18540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ce9f400aad740d7888d440be4982296 [sys.catalog]: This master's current role is: LEADER
I20260927 02:37:30.299073 18541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ce9f400aad740d7888d440be4982296 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ce9f400aad740d7888d440be4982296. Latest consensus state: current_term: 1 leader_uuid: "0ce9f400aad740d7888d440be4982296" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce9f400aad740d7888d440be4982296" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36067 } } }
I20260927 02:37:30.299237 18541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ce9f400aad740d7888d440be4982296 [sys.catalog]: This master's current role is: LEADER
I20260927 02:37:30.299875 18548 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:37:30.302707 18548 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:37:30.307793 18548 catalog_manager.cc:1347] Generated new cluster ID: f2e879e8658a45939ccb08756e670842
I20260927 02:37:30.307858 18548 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:37:30.317898 18548 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:37:30.318691 18548 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:37:30.328226 18548 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ce9f400aad740d7888d440be4982296: Generated new TSK 0
I20260927 02:37:30.328838 18548 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:37:30.363003 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36067
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:37:30.465830 18560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:37:30.466050 18560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:37:30.466089 18560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:37:30.469940 18560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:37:30.469983 18560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:37:30.470036 18560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:37:30.474947 18560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:37:30.475783 18560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:37:30.476636 18560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:37:30.486502 18566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:30.486467 18565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:30.486490 18568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:30.486862 18560 server_base.cc:1034] running on GCE node
I20260927 02:37:30.487275 18560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:37:30.487906 18560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:37:30.489020 18560 hybrid_clock.cc:648] HybridClock initialized: now 1790476650488997 us; error 32 us; skew 500 ppm
Sep 27 02:37:30 dist-test-slave-grlk krb5kdc[18453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:37:30.491616 18560 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:37:30.492442 18560 webserver.cc:460] Webserver started at http://127.0.98.1:37847/ using document root <none> and password file <none>
I20260927 02:37:30.492975 18560 fs_manager.cc:359] Metadata directory not provided
I20260927 02:37:30.493041 18560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:37:30.493212 18560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:37:30.495121 18560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-0/data/instance:
uuid: "edd8bbb579b843e9943261acef18d41d"
format_stamp: "Formatted at 2026-09-27 02:37:30 on dist-test-slave-grlk"
server_key: "26933033ccdbc4952aee2d2ed0f5a8f8"
server_key_iv: "5f55e22f0ab50f28bfc971635bb74931"
server_key_version: "encryptionkey@0"
I20260927 02:37:30.495590 18560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-0/wal/instance:
uuid: "edd8bbb579b843e9943261acef18d41d"
format_stamp: "Formatted at 2026-09-27 02:37:30 on dist-test-slave-grlk"
server_key: "26933033ccdbc4952aee2d2ed0f5a8f8"
server_key_iv: "5f55e22f0ab50f28bfc971635bb74931"
server_key_version: "encryptionkey@0"
I20260927 02:37:30.499012 18560 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:37:30.501178 18575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:30.502272 18560 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 02:37:30.502367 18560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-0/wal
uuid: "edd8bbb579b843e9943261acef18d41d"
format_stamp: "Formatted at 2026-09-27 02:37:30 on dist-test-slave-grlk"
server_key: "26933033ccdbc4952aee2d2ed0f5a8f8"
server_key_iv: "5f55e22f0ab50f28bfc971635bb74931"
server_key_version: "encryptionkey@0"
I20260927 02:37:30.502457 18560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:37:30.528223 18560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:37:30.530834 18560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:37:30.530992 18560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:37:30.531517 18560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:37:30.532264 18560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:37:30.532315 18560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:30.532370 18560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:37:30.532398 18560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:30.540364 18560 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46465
I20260927 02:37:30.540428 18688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46465 every 8 connection(s)
I20260927 02:37:30.541211 18560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:37:30.548295 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 18560
I20260927 02:37:30.548684 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:37:30.549049 392 external_mini_cluster.cc:1428] Setting key 0cb91a19e6f1eebf00c40704fadf82d2
Sep 27 02:37:30 dist-test-slave-grlk krb5kdc[18453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:37:30.553526 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:30.542763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48043 (local address 127.0.98.62:36067)
0927 02:37:30.542887 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:30.542890 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:30.543624 (+ 734us) server_negotiation.cc:407] Connection header received
0927 02:37:30.544818 (+ 1194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:30.544822 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:30.544873 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:30.544963 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:30.546012 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:30.546553 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:30.547618 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:30.547847 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:30.547934 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:30.550486 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:30.550488 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:30.550490 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:37:30.550511 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:30.551691 (+ 1180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:30.552154 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:30.552157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:30.552158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:30.552229 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:30.552526 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:30.552529 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:30.552531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:30.552693 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:30.552775 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:30.553255 (+ 480us) server_negotiation.cc:299] Negotiation successful
0927 02:37:30.553373 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 02:37:30.554235 18691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:30.543097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36067 (local address 127.0.98.1:48043)
0927 02:37:30.543487 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:30.543533 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:30.544631 (+ 1098us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:30.545117 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:30.545124 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:30.545442 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:37:30.545877 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:30.545890 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:30.546663 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:30.546666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:30.547506 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:30.547515 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:30.547975 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:30.547979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:30.548170 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:30.548721 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:37:30.548741 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:37:30.550339 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:37:30.551793 (+ 1454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:37:30.551798 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:37:30.551811 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:37:30.552065 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:37:30.552306 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:37:30.552309 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:37:30.552311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:37:30.552442 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:37:30.552784 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:37:30.552790 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:37:30.553046 (+ 256us) client_negotiation.cc:769] Sending connection context
0927 02:37:30.553203 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 02:37:30.553498 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":106,"threads_started":1}
I20260927 02:37:30.555284 18689 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36067
I20260927 02:37:30.555518 18689 heartbeater.cc:464] Registering TS with master...
I20260927 02:37:30.556031 18689 heartbeater.cc:511] Master 127.0.98.62:36067 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:37:30.557439 18498 ts_manager.cc:194] Registered new tserver with Master: edd8bbb579b843e9943261acef18d41d (127.0.98.1:46465)
I20260927 02:37:30.558463 18498 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48043
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:37:30.571480 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36067
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:37:30.672125 18694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:37:30.672344 18694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:37:30.672382 18694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:37:30.676326 18694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:37:30.676385 18694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:37:30.676440 18694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:37:30.681449 18694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:37:30.682369 18694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:37:30.683290 18694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:37:30.692569 18702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:30.692574 18699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:30.692554 18700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:30.692772 18694 server_base.cc:1034] running on GCE node
I20260927 02:37:30.693234 18694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:37:30.693825 18694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:37:30.694974 18694 hybrid_clock.cc:648] HybridClock initialized: now 1790476650694953 us; error 32 us; skew 500 ppm
Sep 27 02:37:30 dist-test-slave-grlk krb5kdc[18453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:37:30.697436 18694 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:37:30.698254 18694 webserver.cc:460] Webserver started at http://127.0.98.2:42973/ using document root <none> and password file <none>
I20260927 02:37:30.698782 18694 fs_manager.cc:359] Metadata directory not provided
I20260927 02:37:30.698840 18694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:37:30.698992 18694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:37:30.700954 18694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-1/data/instance:
uuid: "3a5ed0e410b54c8ca4b6cb1dcbd81170"
format_stamp: "Formatted at 2026-09-27 02:37:30 on dist-test-slave-grlk"
server_key: "11c30714bab6795968f77f44d7196840"
server_key_iv: "724d02b08bc7abac6080a906ae9db617"
server_key_version: "encryptionkey@0"
I20260927 02:37:30.701453 18694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3a5ed0e410b54c8ca4b6cb1dcbd81170"
format_stamp: "Formatted at 2026-09-27 02:37:30 on dist-test-slave-grlk"
server_key: "11c30714bab6795968f77f44d7196840"
server_key_iv: "724d02b08bc7abac6080a906ae9db617"
server_key_version: "encryptionkey@0"
I20260927 02:37:30.704718 18694 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260927 02:37:30.706899 18709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:30.707994 18694 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 02:37:30.708086 18694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-1/wal
uuid: "3a5ed0e410b54c8ca4b6cb1dcbd81170"
format_stamp: "Formatted at 2026-09-27 02:37:30 on dist-test-slave-grlk"
server_key: "11c30714bab6795968f77f44d7196840"
server_key_iv: "724d02b08bc7abac6080a906ae9db617"
server_key_version: "encryptionkey@0"
I20260927 02:37:30.708177 18694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:37:30.724222 18694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:37:30.726713 18694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:37:30.726866 18694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:37:30.727368 18694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:37:30.728101 18694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:37:30.728144 18694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:30.728184 18694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:37:30.728212 18694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:30.736140 18694 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46657
I20260927 02:37:30.736183 18822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46657 every 8 connection(s)
I20260927 02:37:30.736920 18694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 02:37:30 dist-test-slave-grlk krb5kdc[18453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:37:30.746374 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 18694
I20260927 02:37:30.746498 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:37:30.746836 392 external_mini_cluster.cc:1428] Setting key 3be92d3e909c537342dd556efd33426a
I20260927 02:37:30.749318 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:30.738533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49639 (local address 127.0.98.62:36067)
0927 02:37:30.738667 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:30.738670 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:30.739386 (+ 716us) server_negotiation.cc:407] Connection header received
0927 02:37:30.740577 (+ 1191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:30.740579 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:30.740623 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:30.740690 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:30.741768 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:30.742270 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:30.743371 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:30.743577 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:30.743645 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:30.745869 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:30.745872 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:30.745874 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:37:30.745897 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:30.747313 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:30.747851 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:30.747854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:30.747856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:30.747941 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:30.748244 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:30.748246 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:30.748248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:30.748422 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:30.748496 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:30.749071 (+ 575us) server_negotiation.cc:299] Negotiation successful
0927 02:37:30.749184 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 02:37:30.750167 18825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:30.738824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36067 (local address 127.0.98.2:49639)
0927 02:37:30.739236 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:30.739275 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:30.740397 (+ 1122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:30.740831 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:30.740837 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:30.741214 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:37:30.741645 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:30.741658 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:30.742381 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:30.742384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:30.743272 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:30.743281 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:30.743724 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:30.743728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:30.743876 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:30.744344 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:37:30.744359 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:37:30.745713 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:37:30.747421 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:37:30.747426 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:37:30.747439 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:37:30.747748 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:37:30.748019 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:37:30.748022 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:37:30.748024 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:37:30.748149 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:37:30.748521 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:37:30.748527 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:37:30.748810 (+ 283us) client_negotiation.cc:769] Sending connection context
0927 02:37:30.749016 (+ 206us) client_negotiation.cc:240] Negotiation successful
0927 02:37:30.749308 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":106,"threads_started":1}
I20260927 02:37:30.751250 18823 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36067
I20260927 02:37:30.751448 18823 heartbeater.cc:464] Registering TS with master...
I20260927 02:37:30.751857 18823 heartbeater.cc:511] Master 127.0.98.62:36067 requested a full tablet report, sending...
I20260927 02:37:30.752853 18498 ts_manager.cc:194] Registered new tserver with Master: 3a5ed0e410b54c8ca4b6cb1dcbd81170 (127.0.98.2:46657)
I20260927 02:37:30.753580 18498 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49639
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:37:30.770970 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36067
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:37:30.874708 18828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:37:30.874955 18828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:37:30.874994 18828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:37:30.878845 18828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:37:30.878897 18828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:37:30.878952 18828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:37:30.884093 18828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:37:30.885038 18828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:37:30.885934 18828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:37:30.894681 18834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:30.894841 18828 server_base.cc:1034] running on GCE node
W20260927 02:37:30.894971 18836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:30.894879 18833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:30.895403 18828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:37:30.895992 18828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:37:30.897145 18828 hybrid_clock.cc:648] HybridClock initialized: now 1790476650897131 us; error 24 us; skew 500 ppm
Sep 27 02:37:30 dist-test-slave-grlk krb5kdc[18453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:37:30.903412 18828 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:37:30.904239 18828 webserver.cc:460] Webserver started at http://127.0.98.3:39661/ using document root <none> and password file <none>
I20260927 02:37:30.904747 18828 fs_manager.cc:359] Metadata directory not provided
I20260927 02:37:30.904808 18828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:37:30.905025 18828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:37:30.907110 18828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-2/data/instance:
uuid: "8688c5bc07504ae4ab72f4abc596e1b5"
format_stamp: "Formatted at 2026-09-27 02:37:30 on dist-test-slave-grlk"
server_key: "c441cc9e99d67f116c854cdb746ba7f5"
server_key_iv: "3ce3565328cfdf5088c8dad0b8df98ba"
server_key_version: "encryptionkey@0"
I20260927 02:37:30.907611 18828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8688c5bc07504ae4ab72f4abc596e1b5"
format_stamp: "Formatted at 2026-09-27 02:37:30 on dist-test-slave-grlk"
server_key: "c441cc9e99d67f116c854cdb746ba7f5"
server_key_iv: "3ce3565328cfdf5088c8dad0b8df98ba"
server_key_version: "encryptionkey@0"
I20260927 02:37:30.911167 18828 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:37:30.913561 18843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:30.914722 18828 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:37:30.914832 18828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-2/wal
uuid: "8688c5bc07504ae4ab72f4abc596e1b5"
format_stamp: "Formatted at 2026-09-27 02:37:30 on dist-test-slave-grlk"
server_key: "c441cc9e99d67f116c854cdb746ba7f5"
server_key_iv: "3ce3565328cfdf5088c8dad0b8df98ba"
server_key_version: "encryptionkey@0"
I20260927 02:37:30.914922 18828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:37:30.931589 18828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:37:30.934513 18828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:37:30.934679 18828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:37:30.935209 18828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:37:30.935943 18828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:37:30.935987 18828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:30.936035 18828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:37:30.936064 18828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:30.944053 18828 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33953
I20260927 02:37:30.944120 18956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33953 every 8 connection(s)
I20260927 02:37:30.944864 18828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:37:30.945976 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 18828
I20260927 02:37:30.946105 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:37:30.946424 392 external_mini_cluster.cc:1428] Setting key ee6be6b4b3fc553b46af66f15e418ddf
Sep 27 02:37:30 dist-test-slave-grlk krb5kdc[18453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:37:30.956741 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:30.946566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48181 (local address 127.0.98.62:36067)
0927 02:37:30.946687 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:30.946692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:30.947639 (+ 947us) server_negotiation.cc:407] Connection header received
0927 02:37:30.948872 (+ 1233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:30.948875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:30.948923 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:30.949034 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:30.950011 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:30.950528 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:30.951575 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:30.951791 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:30.951887 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:30.954086 (+ 2199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:30.954090 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:30.954092 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:37:30.954114 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:30.955202 (+ 1088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:30.955616 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:30.955619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:30.955620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:30.955688 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:30.955935 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:30.955937 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:30.955938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:30.956090 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:30.956159 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:30.956540 (+ 381us) server_negotiation.cc:299] Negotiation successful
0927 02:37:30.956630 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 02:37:30.957470 18959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:30.947049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36067 (local address 127.0.98.3:48181)
0927 02:37:30.947513 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:30.947559 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:30.948697 (+ 1138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:30.949153 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:30.949159 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:30.949475 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:37:30.949906 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:30.949918 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:30.950620 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:30.950622 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:30.951455 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:30.951463 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:30.951913 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:30.951916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:30.952049 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:30.952526 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:37:30.952542 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:37:30.953951 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:37:30.955293 (+ 1342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:37:30.955298 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:37:30.955308 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:37:30.955535 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:37:30.955753 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:37:30.955755 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:37:30.955757 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:37:30.955873 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:37:30.956158 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:37:30.956163 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:37:30.956367 (+ 204us) client_negotiation.cc:769] Sending connection context
0927 02:37:30.956504 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 02:37:30.956718 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":124,"threads_started":1}
I20260927 02:37:30.958395 18957 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36067
I20260927 02:37:30.958607 18957 heartbeater.cc:464] Registering TS with master...
I20260927 02:37:30.959111 18957 heartbeater.cc:511] Master 127.0.98.62:36067 requested a full tablet report, sending...
I20260927 02:37:30.960080 18498 ts_manager.cc:194] Registered new tserver with Master: 8688c5bc07504ae4ab72f4abc596e1b5 (127.0.98.3:33953)
I20260927 02:37:30.960660 18498 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48181
I20260927 02:37:30.970954 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:37:30.979032 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:30.972743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33284 (local address 127.0.98.62:36067)
0927 02:37:30.972871 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:30.972874 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:30.973082 (+ 208us) server_negotiation.cc:407] Connection header received
0927 02:37:30.973179 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:30.973181 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:30.973216 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:30.973269 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:30.973830 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:30.974298 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:30.975109 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:30.975325 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:30.975395 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:30.976674 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:30.976677 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:30.976679 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:37:30.976693 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:30.977770 (+ 1077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:30.978158 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:30.978160 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:30.978161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:30.978220 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:30.978428 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:30.978430 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:30.978431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:30.978586 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:30.978680 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:30.978868 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 02:37:30.978932 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 02:37:30.981361 18498 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33284:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:37:30.984141 18498 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:37:30.996791 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:30.991926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46465 (local address 127.0.0.1:55422)
0927 02:37:30.992571 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:30.992583 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:30.992649 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:30.993286 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:30.993288 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:30.993307 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:37:30.993379 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:30.993384 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:30.994584 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:30.994587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:30.995566 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:30.995576 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:30.996303 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:30.996307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:30.996406 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:30.996438 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 02:37:30.996535 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 02:37:30.996643 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"mutex_wait_us":30,"thread_start_us":79,"threads_started":1}
I20260927 02:37:30.997115 18970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:30.991612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46657 (local address 127.0.0.1:58890)
0927 02:37:30.991882 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:30.991892 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:30.991956 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:30.992922 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:30.992925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:30.992963 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:37:30.993052 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:30.993058 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:30.994093 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:30.994096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:30.995468 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:30.995479 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:30.996537 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:30.996540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:30.996759 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:30.996793 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 02:37:30.996855 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 02:37:30.996980 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":157,"thread_start_us":86,"threads_started":1}
I20260927 02:37:30.997304 18969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:30.991439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58272 (local address 127.0.98.3:33953)
0927 02:37:30.992128 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:30.992139 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:30.992160 (+ 21us) server_negotiation.cc:407] Connection header received
0927 02:37:30.992230 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:30.992235 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:30.992432 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:30.992919 (+ 487us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:37:30.993085 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:30.994202 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:30.995631 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:30.996391 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:30.996529 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:30.996657 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:30.997059 (+ 402us) server_negotiation.cc:299] Negotiation successful
0927 02:37:30.997152 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":118,"threads_started":1}
I20260927 02:37:30.997339 18973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:30.992242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55422 (local address 127.0.98.1:46465)
0927 02:37:30.992862 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:30.992873 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:30.992891 (+ 18us) server_negotiation.cc:407] Connection header received
0927 02:37:30.992979 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:30.992985 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:30.993151 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:30.993301 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:37:30.993493 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:30.994474 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:30.995680 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:30.996191 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:30.996293 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:30.996429 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:30.997128 (+ 699us) server_negotiation.cc:299] Negotiation successful
0927 02:37:30.997209 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":87,"threads_started":1}
I20260927 02:37:30.997417 18968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:30.991279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33953 (local address 127.0.0.1:58272)
0927 02:37:30.991677 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:30.991723 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:30.991848 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:30.992601 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:30.992608 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:30.992632 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:37:30.992775 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:30.992781 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:30.994326 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:30.994329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:30.995520 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:30.995527 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:30.996546 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:30.996550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:30.996738 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:30.996787 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 02:37:30.996857 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 02:37:30.996964 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":89,"thread_start_us":66,"threads_started":1}
I20260927 02:37:30.997676 18972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:30.991936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58890 (local address 127.0.98.2:46657)
0927 02:37:30.992479 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:30.992490 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:30.992511 (+ 21us) server_negotiation.cc:407] Connection header received
0927 02:37:30.992579 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:30.992585 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:30.992771 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:30.992967 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:37:30.993164 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:30.993982 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:30.995600 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:30.996391 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:30.996736 (+ 345us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:30.997104 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:30.997156 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 02:37:30.997533 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":122,"threads_started":1}
I20260927 02:37:30.999955 18623 tablet_service.cc:1467] Processing CreateTablet for tablet 879a52471b024c5aa7bd8d1ee67d2fa4 (DEFAULT_TABLE table=test-table [id=c2092904c1334e0ca6bb55290dfda313]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:37:31.000298 18891 tablet_service.cc:1467] Processing CreateTablet for tablet 879a52471b024c5aa7bd8d1ee67d2fa4 (DEFAULT_TABLE table=test-table [id=c2092904c1334e0ca6bb55290dfda313]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:37:31.000203 18757 tablet_service.cc:1467] Processing CreateTablet for tablet 879a52471b024c5aa7bd8d1ee67d2fa4 (DEFAULT_TABLE table=test-table [id=c2092904c1334e0ca6bb55290dfda313]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:37:31.000901 18623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 879a52471b024c5aa7bd8d1ee67d2fa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:37:31.001190 18891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 879a52471b024c5aa7bd8d1ee67d2fa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:37:31.001190 18757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 879a52471b024c5aa7bd8d1ee67d2fa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:37:31.006767 18974 tablet_bootstrap.cc:492] T 879a52471b024c5aa7bd8d1ee67d2fa4 P edd8bbb579b843e9943261acef18d41d: Bootstrap starting.
I20260927 02:37:31.008206 18975 tablet_bootstrap.cc:492] T 879a52471b024c5aa7bd8d1ee67d2fa4 P 3a5ed0e410b54c8ca4b6cb1dcbd81170: Bootstrap starting.
I20260927 02:37:31.008486 18976 tablet_bootstrap.cc:492] T 879a52471b024c5aa7bd8d1ee67d2fa4 P 8688c5bc07504ae4ab72f4abc596e1b5: Bootstrap starting.
I20260927 02:37:31.009500 18974 tablet_bootstrap.cc:654] T 879a52471b024c5aa7bd8d1ee67d2fa4 P edd8bbb579b843e9943261acef18d41d: Neither blocks nor log segments found. Creating new log.
I20260927 02:37:31.010272 18975 tablet_bootstrap.cc:654] T 879a52471b024c5aa7bd8d1ee67d2fa4 P 3a5ed0e410b54c8ca4b6cb1dcbd81170: Neither blocks nor log segments found. Creating new log.
I20260927 02:37:31.010272 18976 tablet_bootstrap.cc:654] T 879a52471b024c5aa7bd8d1ee67d2fa4 P 8688c5bc07504ae4ab72f4abc596e1b5: Neither blocks nor log segments found. Creating new log.
I20260927 02:37:31.010476 18974 log.cc:824] T 879a52471b024c5aa7bd8d1ee67d2fa4 P edd8bbb579b843e9943261acef18d41d: Log is configured to *not* fsync() on all Append() calls
I20260927 02:37:31.011436 18976 log.cc:824] T 879a52471b024c5aa7bd8d1ee67d2fa4 P 8688c5bc07504ae4ab72f4abc596e1b5: Log is configured to *not* fsync() on all Append() calls
I20260927 02:37:31.011436 18975 log.cc:824] T 879a52471b024c5aa7bd8d1ee67d2fa4 P 3a5ed0e410b54c8ca4b6cb1dcbd81170: Log is configured to *not* fsync() on all Append() calls
I20260927 02:37:31.012317 18974 tablet_bootstrap.cc:492] T 879a52471b024c5aa7bd8d1ee67d2fa4 P edd8bbb579b843e9943261acef18d41d: No bootstrap required, opened a new log
I20260927 02:37:31.012517 18974 ts_tablet_manager.cc:1397] T 879a52471b024c5aa7bd8d1ee67d2fa4 P edd8bbb579b843e9943261acef18d41d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 02:37:31.013265 18976 tablet_bootstrap.cc:492] T 879a52471b024c5aa7bd8d1ee67d2fa4 P 8688c5bc07504ae4ab72f4abc596e1b5: No bootstrap required, opened a new log
I20260927 02:37:31.013497 18976 ts_tablet_manager.cc:1397] T 879a52471b024c5aa7bd8d1ee67d2fa4 P 8688c5bc07504ae4ab72f4abc596e1b5: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 02:37:31.013636 18975 tablet_bootstrap.cc:492] T 879a52471b024c5aa7bd8d1ee67d2fa4 P 3a5ed0e410b54c8ca4b6cb1dcbd81170: No bootstrap required, opened a new log
I20260927 02:37:31.013797 18975 ts_tablet_manager.cc:1397] T 879a52471b024c5aa7bd8d1ee67d2fa4 P 3a5ed0e410b54c8ca4b6cb1dcbd81170: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 02:37:31.015466 18974 raft_consensus.cc:348] T 879a52471b024c5aa7bd8d1ee67d2fa4 P edd8bbb579b843e9943261acef18d41d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8688c5bc07504ae4ab72f4abc596e1b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33953 } } peers { permanent_uuid: "edd8bbb579b843e9943261acef18d41d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46465 } } peers { permanent_uuid: "3a5ed0e410b54c8ca4b6cb1dcbd81170" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46657 } }
I20260927 02:37:31.015664 18974 raft_consensus.cc:374] T 879a52471b024c5aa7bd8d1ee67d2fa4 P edd8bbb579b843e9943261acef18d41d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:37:31.015730 18974 raft_consensus.cc:729] T 879a52471b024c5aa7bd8d1ee67d2fa4 P edd8bbb579b843e9943261acef18d41d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edd8bbb579b843e9943261acef18d41d, State: Initialized, Role: FOLLOWER
I20260927 02:37:31.016222 18974 consensus_queue.cc:260] T 879a52471b024c5aa7bd8d1ee67d2fa4 P edd8bbb579b843e9943261acef18d41d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8688c5bc07504ae4ab72f4abc596e1b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33953 } } peers { permanent_uuid: "edd8bbb579b843e9943261acef18d41d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46465 } } peers { permanent_uuid: "3a5ed0e410b54c8ca4b6cb1dcbd81170" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46657 } }
I20260927 02:37:31.017180 18974 ts_tablet_manager.cc:1428] T 879a52471b024c5aa7bd8d1ee67d2fa4 P edd8bbb579b843e9943261acef18d41d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 02:37:31.017145 18976 raft_consensus.cc:348] T 879a52471b024c5aa7bd8d1ee67d2fa4 P 8688c5bc07504ae4ab72f4abc596e1b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8688c5bc07504ae4ab72f4abc596e1b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33953 } } peers { permanent_uuid: "edd8bbb579b843e9943261acef18d41d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46465 } } peers { permanent_uuid: "3a5ed0e410b54c8ca4b6cb1dcbd81170" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46657 } }
I20260927 02:37:31.017194 18975 raft_consensus.cc:348] T 879a52471b024c5aa7bd8d1ee67d2fa4 P 3a5ed0e410b54c8ca4b6cb1dcbd81170 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8688c5bc07504ae4ab72f4abc596e1b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33953 } } peers { permanent_uuid: "edd8bbb579b843e9943261acef18d41d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46465 } } peers { permanent_uuid: "3a5ed0e410b54c8ca4b6cb1dcbd81170" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46657 } }
I20260927 02:37:31.017398 18976 raft_consensus.cc:374] T 879a52471b024c5aa7bd8d1ee67d2fa4 P 8688c5bc07504ae4ab72f4abc596e1b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:37:31.017398 18975 raft_consensus.cc:374] T 879a52471b024c5aa7bd8d1ee67d2fa4 P 3a5ed0e410b54c8ca4b6cb1dcbd81170 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:37:31.017444 18975 raft_consensus.cc:729] T 879a52471b024c5aa7bd8d1ee67d2fa4 P 3a5ed0e410b54c8ca4b6cb1dcbd81170 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a5ed0e410b54c8ca4b6cb1dcbd81170, State: Initialized, Role: FOLLOWER
I20260927 02:37:31.017450 18976 raft_consensus.cc:729] T 879a52471b024c5aa7bd8d1ee67d2fa4 P 8688c5bc07504ae4ab72f4abc596e1b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8688c5bc07504ae4ab72f4abc596e1b5, State: Initialized, Role: FOLLOWER
I20260927 02:37:31.017499 18689 heartbeater.cc:503] Master 127.0.98.62:36067 was elected leader, sending a full tablet report...
I20260927 02:37:31.017839 18975 consensus_queue.cc:260] T 879a52471b024c5aa7bd8d1ee67d2fa4 P 3a5ed0e410b54c8ca4b6cb1dcbd81170 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8688c5bc07504ae4ab72f4abc596e1b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33953 } } peers { permanent_uuid: "edd8bbb579b843e9943261acef18d41d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46465 } } peers { permanent_uuid: "3a5ed0e410b54c8ca4b6cb1dcbd81170" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46657 } }
I20260927 02:37:31.017839 18976 consensus_queue.cc:260] T 879a52471b024c5aa7bd8d1ee67d2fa4 P 8688c5bc07504ae4ab72f4abc596e1b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8688c5bc07504ae4ab72f4abc596e1b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33953 } } peers { permanent_uuid: "edd8bbb579b843e9943261acef18d41d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46465 } } peers { permanent_uuid: "3a5ed0e410b54c8ca4b6cb1dcbd81170" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46657 } }
I20260927 02:37:31.018688 18957 heartbeater.cc:503] Master 127.0.98.62:36067 was elected leader, sending a full tablet report...
I20260927 02:37:31.018699 18975 ts_tablet_manager.cc:1428] T 879a52471b024c5aa7bd8d1ee67d2fa4 P 3a5ed0e410b54c8ca4b6cb1dcbd81170: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 02:37:31.018679 18823 heartbeater.cc:503] Master 127.0.98.62:36067 was elected leader, sending a full tablet report...
I20260927 02:37:31.020471 18976 ts_tablet_manager.cc:1428] T 879a52471b024c5aa7bd8d1ee67d2fa4 P 8688c5bc07504ae4ab72f4abc596e1b5: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
W20260927 02:37:31.042146 18690 tablet.cc:2369] T 879a52471b024c5aa7bd8d1ee67d2fa4 P edd8bbb579b843e9943261acef18d41d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:37:31.162629 18980 raft_consensus.cc:482] T 879a52471b024c5aa7bd8d1ee67d2fa4 P edd8bbb579b843e9943261acef18d41d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:37:31.162863 18980 raft_consensus.cc:504] T 879a52471b024c5aa7bd8d1ee67d2fa4 P edd8bbb579b843e9943261acef18d41d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8688c5bc07504ae4ab72f4abc596e1b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33953 } } peers { permanent_uuid: "edd8bbb579b843e9943261acef18d41d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46465 } } peers { permanent_uuid: "3a5ed0e410b54c8ca4b6cb1dcbd81170" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46657 } }
I20260927 02:37:31.163940 18980 leader_election.cc:288] T 879a52471b024c5aa7bd8d1ee67d2fa4 P edd8bbb579b843e9943261acef18d41d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8688c5bc07504ae4ab72f4abc596e1b5 (127.0.98.3:33953), 3a5ed0e410b54c8ca4b6cb1dcbd81170 (127.0.98.2:46657)
I20260927 02:37:31.167563 18969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:31.164528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49913 (local address 127.0.98.3:33953)
0927 02:37:31.164670 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:31.164673 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:31.164684 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:37:31.164726 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:31.164728 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:31.164765 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:31.164826 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:37:31.165111 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:31.165606 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:31.166673 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:31.167063 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:31.167153 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:31.167213 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:31.167393 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 02:37:31.167458 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 02:37:31.167578 18983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:31.164191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33953 (local address 127.0.98.1:49913)
0927 02:37:31.164457 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:31.164473 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:31.164559 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:31.164851 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:31.164854 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:31.164870 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:37:31.165006 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:31.165012 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:31.165707 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:31.165710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:31.166556 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:31.166564 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:31.167169 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:31.167172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:31.167284 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:31.167302 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 02:37:31.167365 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 02:37:31.167430 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":42,"thread_start_us":105,"threads_started":1}
I20260927 02:37:31.167657 18972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:31.164302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42953 (local address 127.0.98.2:46657)
0927 02:37:31.164428 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:31.164432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:31.164693 (+ 261us) server_negotiation.cc:407] Connection header received
0927 02:37:31.164820 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:31.164822 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:31.164863 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:31.164918 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:37:31.165207 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:31.165732 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:31.166751 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:31.167138 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:31.167227 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:31.167283 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:31.167463 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 02:37:31.167528 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 02:37:31.167836 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:31.164205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46657 (local address 127.0.98.1:42953)
0927 02:37:31.164596 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:31.164609 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:31.164713 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:31.164966 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:31.164969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:31.164984 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:37:31.165099 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:31.165104 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:31.165836 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:31.165840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:31.166642 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:31.166650 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:31.167244 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:31.167248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:31.167362 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:31.167379 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 02:37:31.167440 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 02:37:31.167506 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":17,"thread_start_us":105,"threads_started":1}
I20260927 02:37:31.168380 18777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879a52471b024c5aa7bd8d1ee67d2fa4" candidate_uuid: "edd8bbb579b843e9943261acef18d41d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a5ed0e410b54c8ca4b6cb1dcbd81170" is_pre_election: true
I20260927 02:37:31.168382 18911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879a52471b024c5aa7bd8d1ee67d2fa4" candidate_uuid: "edd8bbb579b843e9943261acef18d41d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8688c5bc07504ae4ab72f4abc596e1b5" is_pre_election: true
I20260927 02:37:31.168671 18911 raft_consensus.cc:2454] T 879a52471b024c5aa7bd8d1ee67d2fa4 P 8688c5bc07504ae4ab72f4abc596e1b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edd8bbb579b843e9943261acef18d41d in term 0.
I20260927 02:37:31.168671 18777 raft_consensus.cc:2454] T 879a52471b024c5aa7bd8d1ee67d2fa4 P 3a5ed0e410b54c8ca4b6cb1dcbd81170 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edd8bbb579b843e9943261acef18d41d in term 0.
I20260927 02:37:31.169111 18577 leader_election.cc:302] T 879a52471b024c5aa7bd8d1ee67d2fa4 P edd8bbb579b843e9943261acef18d41d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a5ed0e410b54c8ca4b6cb1dcbd81170, edd8bbb579b843e9943261acef18d41d; no voters:
I20260927 02:37:31.169327 18980 raft_consensus.cc:2781] T 879a52471b024c5aa7bd8d1ee67d2fa4 P edd8bbb579b843e9943261acef18d41d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:37:31.169402 18980 raft_consensus.cc:482] T 879a52471b024c5aa7bd8d1ee67d2fa4 P edd8bbb579b843e9943261acef18d41d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:37:31.169452 18980 raft_consensus.cc:3037] T 879a52471b024c5aa7bd8d1ee67d2fa4 P edd8bbb579b843e9943261acef18d41d [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:37:31.170423 18980 raft_consensus.cc:504] T 879a52471b024c5aa7bd8d1ee67d2fa4 P edd8bbb579b843e9943261acef18d41d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8688c5bc07504ae4ab72f4abc596e1b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33953 } } peers { permanent_uuid: "edd8bbb579b843e9943261acef18d41d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46465 } } peers { permanent_uuid: "3a5ed0e410b54c8ca4b6cb1dcbd81170" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46657 } }
I20260927 02:37:31.170747 18980 leader_election.cc:288] T 879a52471b024c5aa7bd8d1ee67d2fa4 P edd8bbb579b843e9943261acef18d41d [CANDIDATE]: Term 1 election: Requested vote from peers 8688c5bc07504ae4ab72f4abc596e1b5 (127.0.98.3:33953), 3a5ed0e410b54c8ca4b6cb1dcbd81170 (127.0.98.2:46657)
I20260927 02:37:31.171074 18911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879a52471b024c5aa7bd8d1ee67d2fa4" candidate_uuid: "edd8bbb579b843e9943261acef18d41d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8688c5bc07504ae4ab72f4abc596e1b5"
I20260927 02:37:31.171082 18777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879a52471b024c5aa7bd8d1ee67d2fa4" candidate_uuid: "edd8bbb579b843e9943261acef18d41d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a5ed0e410b54c8ca4b6cb1dcbd81170"
I20260927 02:37:31.171197 18777 raft_consensus.cc:3037] T 879a52471b024c5aa7bd8d1ee67d2fa4 P 3a5ed0e410b54c8ca4b6cb1dcbd81170 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:37:31.171197 18911 raft_consensus.cc:3037] T 879a52471b024c5aa7bd8d1ee67d2fa4 P 8688c5bc07504ae4ab72f4abc596e1b5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:37:31.172405 18911 raft_consensus.cc:2454] T 879a52471b024c5aa7bd8d1ee67d2fa4 P 8688c5bc07504ae4ab72f4abc596e1b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edd8bbb579b843e9943261acef18d41d in term 1.
I20260927 02:37:31.172405 18777 raft_consensus.cc:2454] T 879a52471b024c5aa7bd8d1ee67d2fa4 P 3a5ed0e410b54c8ca4b6cb1dcbd81170 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edd8bbb579b843e9943261acef18d41d in term 1.
I20260927 02:37:31.172688 18577 leader_election.cc:302] T 879a52471b024c5aa7bd8d1ee67d2fa4 P edd8bbb579b843e9943261acef18d41d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a5ed0e410b54c8ca4b6cb1dcbd81170, edd8bbb579b843e9943261acef18d41d; no voters:
I20260927 02:37:31.172847 18980 raft_consensus.cc:2781] T 879a52471b024c5aa7bd8d1ee67d2fa4 P edd8bbb579b843e9943261acef18d41d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:37:31.173075 18980 raft_consensus.cc:686] T 879a52471b024c5aa7bd8d1ee67d2fa4 P edd8bbb579b843e9943261acef18d41d [term 1 LEADER]: Becoming Leader. State: Replica: edd8bbb579b843e9943261acef18d41d, State: Running, Role: LEADER
I20260927 02:37:31.173415 18980 consensus_queue.cc:237] T 879a52471b024c5aa7bd8d1ee67d2fa4 P edd8bbb579b843e9943261acef18d41d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8688c5bc07504ae4ab72f4abc596e1b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33953 } } peers { permanent_uuid: "edd8bbb579b843e9943261acef18d41d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46465 } } peers { permanent_uuid: "3a5ed0e410b54c8ca4b6cb1dcbd81170" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46657 } }
I20260927 02:37:31.176417 18496 catalog_manager.cc:5495] T 879a52471b024c5aa7bd8d1ee67d2fa4 P edd8bbb579b843e9943261acef18d41d reported cstate change: term changed from 0 to 1, leader changed from <none> to edd8bbb579b843e9943261acef18d41d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "edd8bbb579b843e9943261acef18d41d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8688c5bc07504ae4ab72f4abc596e1b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "edd8bbb579b843e9943261acef18d41d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a5ed0e410b54c8ca4b6cb1dcbd81170" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46657 } health_report { overall_health: UNKNOWN } } }
W20260927 02:37:31.195910 18958 tablet.cc:2369] T 879a52471b024c5aa7bd8d1ee67d2fa4 P 8688c5bc07504ae4ab72f4abc596e1b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:37:31.229394 18973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:31.226135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55432 (local address 127.0.98.1:46465)
0927 02:37:31.226266 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:31.226271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:31.226296 (+ 25us) server_negotiation.cc:407] Connection header received
0927 02:37:31.226365 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:31.226367 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:31.226417 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:31.226501 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:37:31.226717 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:31.227211 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:31.228012 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:31.228217 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:31.228300 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:31.228594 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:37:31.229018 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:37:31.229154 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:31.229232 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 02:37:31.229270 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 02:37:31.234752 18911 raft_consensus.cc:1261] T 879a52471b024c5aa7bd8d1ee67d2fa4 P 8688c5bc07504ae4ab72f4abc596e1b5 [term 1 FOLLOWER]: Refusing update from remote peer edd8bbb579b843e9943261acef18d41d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:37:31.234752 18777 raft_consensus.cc:1261] T 879a52471b024c5aa7bd8d1ee67d2fa4 P 3a5ed0e410b54c8ca4b6cb1dcbd81170 [term 1 FOLLOWER]: Refusing update from remote peer edd8bbb579b843e9943261acef18d41d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:37:31.235348 18985 consensus_queue.cc:1035] T 879a52471b024c5aa7bd8d1ee67d2fa4 P edd8bbb579b843e9943261acef18d41d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8688c5bc07504ae4ab72f4abc596e1b5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:37:31.235522 18980 consensus_queue.cc:1035] T 879a52471b024c5aa7bd8d1ee67d2fa4 P edd8bbb579b843e9943261acef18d41d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a5ed0e410b54c8ca4b6cb1dcbd81170" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 02:37:31.237921 18824 tablet.cc:2369] T 879a52471b024c5aa7bd8d1ee67d2fa4 P 3a5ed0e410b54c8ca4b6cb1dcbd81170: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:37:31.241179 18988 mvcc.cc:205] Tried to move back new op lower bound from 7333792363451187200 to 7333792363208552448. Current Snapshot: MvccSnapshot[applied={T|T < 7333792363451187200}]
I20260927 02:37:31.249161 18495 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:37:31.249310 18495 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:37:31.251392 18495 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0ce9f400aad740d7888d440be4982296: Sending DeleteTablet for 3 replicas of tablet 879a52471b024c5aa7bd8d1ee67d2fa4
I20260927 02:37:31.252075 18757 tablet_service.cc:1514] Processing DeleteTablet for tablet 879a52471b024c5aa7bd8d1ee67d2fa4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:37:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58890
I20260927 02:37:31.252111 18891 tablet_service.cc:1514] Processing DeleteTablet for tablet 879a52471b024c5aa7bd8d1ee67d2fa4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:37:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58272
I20260927 02:37:31.252210 18623 tablet_service.cc:1514] Processing DeleteTablet for tablet 879a52471b024c5aa7bd8d1ee67d2fa4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:37:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55422
I20260927 02:37:31.252559 18998 tablet_replica.cc:330] T 879a52471b024c5aa7bd8d1ee67d2fa4 P 3a5ed0e410b54c8ca4b6cb1dcbd81170: stopping tablet replica
I20260927 02:37:31.252569 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 18560
I20260927 02:37:31.252619 18999 tablet_replica.cc:330] T 879a52471b024c5aa7bd8d1ee67d2fa4 P 8688c5bc07504ae4ab72f4abc596e1b5: stopping tablet replica
I20260927 02:37:31.252902 18999 raft_consensus.cc:2229] T 879a52471b024c5aa7bd8d1ee67d2fa4 P 8688c5bc07504ae4ab72f4abc596e1b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:37:31.252902 18998 raft_consensus.cc:2229] T 879a52471b024c5aa7bd8d1ee67d2fa4 P 3a5ed0e410b54c8ca4b6cb1dcbd81170 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:37:31.253247 18999 raft_consensus.cc:2258] T 879a52471b024c5aa7bd8d1ee67d2fa4 P 8688c5bc07504ae4ab72f4abc596e1b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:37:31.253247 18998 raft_consensus.cc:2258] T 879a52471b024c5aa7bd8d1ee67d2fa4 P 3a5ed0e410b54c8ca4b6cb1dcbd81170 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:37:31.254724 18998 ts_tablet_manager.cc:1905] T 879a52471b024c5aa7bd8d1ee67d2fa4 P 3a5ed0e410b54c8ca4b6cb1dcbd81170: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:37:31.254724 18999 ts_tablet_manager.cc:1905] T 879a52471b024c5aa7bd8d1ee67d2fa4 P 8688c5bc07504ae4ab72f4abc596e1b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:37:31.257326 18998 ts_tablet_manager.cc:1918] T 879a52471b024c5aa7bd8d1ee67d2fa4 P 3a5ed0e410b54c8ca4b6cb1dcbd81170: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:37:31.257326 18999 ts_tablet_manager.cc:1918] T 879a52471b024c5aa7bd8d1ee67d2fa4 P 8688c5bc07504ae4ab72f4abc596e1b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:37:31.257423 18999 log.cc:1196] T 879a52471b024c5aa7bd8d1ee67d2fa4 P 8688c5bc07504ae4ab72f4abc596e1b5: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-2/wal/wals/879a52471b024c5aa7bd8d1ee67d2fa4
I20260927 02:37:31.257423 18998 log.cc:1196] T 879a52471b024c5aa7bd8d1ee67d2fa4 P 3a5ed0e410b54c8ca4b6cb1dcbd81170: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476410933508-392-0/minicluster-data/ts-1/wal/wals/879a52471b024c5aa7bd8d1ee67d2fa4
I20260927 02:37:31.257737 18999 ts_tablet_manager.cc:1939] T 879a52471b024c5aa7bd8d1ee67d2fa4 P 8688c5bc07504ae4ab72f4abc596e1b5: Deleting consensus metadata
I20260927 02:37:31.257737 18998 ts_tablet_manager.cc:1939] T 879a52471b024c5aa7bd8d1ee67d2fa4 P 3a5ed0e410b54c8ca4b6cb1dcbd81170: Deleting consensus metadata
I20260927 02:37:31.258906 18485 catalog_manager.cc:4841] TS 8688c5bc07504ae4ab72f4abc596e1b5 (127.0.98.3:33953): tablet 879a52471b024c5aa7bd8d1ee67d2fa4 (table test-table [id=c2092904c1334e0ca6bb55290dfda313]) successfully deleted
I20260927 02:37:31.259078 18483 catalog_manager.cc:4841] TS 3a5ed0e410b54c8ca4b6cb1dcbd81170 (127.0.98.2:46657): tablet 879a52471b024c5aa7bd8d1ee67d2fa4 (table test-table [id=c2092904c1334e0ca6bb55290dfda313]) successfully deleted
W20260927 02:37:31.259254 18485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46465 (error 108)
I20260927 02:37:31.259471 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 18694
I20260927 02:37:31.260481 18968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:31.260102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46465 (local address 127.0.0.1:55440)
0927 02:37:31.260332 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:31.260399 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
W20260927 02:37:31.260617 18485 catalog_manager.cc:4584] TS edd8bbb579b843e9943261acef18d41d (127.0.98.1:46465): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 879a52471b024c5aa7bd8d1ee67d2fa4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46465: connect: Connection refused (error 111)
I20260927 02:37:31.264158 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 18828
I20260927 02:37:31.268337 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 18466
2026-09-27T02:37:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2215 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T02:37:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:37:31Z Disabled control of system clock
I20260927 02:37:31.297958 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45259
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 02:37:31.400616 19005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:37:31.400826 19005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:37:31.400858 19005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:37:31.404774 19005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:37:31.404825 19005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:37:31.404838 19005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:37:31.404853 19005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:37:31.404867 19005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:37:31.409416 19005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45259
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:37:31.410262 19005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:37:31.411450 19005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:37:31.419152 19013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:31.419147 19010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:31.419164 19011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:31.419245 19005 server_base.cc:1034] running on GCE node
I20260927 02:37:31.419972 19005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:37:31.421036 19005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:37:31.422204 19005 hybrid_clock.cc:648] HybridClock initialized: now 1790476651422174 us; error 39 us; skew 500 ppm
I20260927 02:37:31.423895 19005 webserver.cc:460] Webserver started at http://127.0.98.62:39289/ using document root <none> and password file <none>
I20260927 02:37:31.424410 19005 fs_manager.cc:359] Metadata directory not provided
I20260927 02:37:31.424466 19005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:37:31.424654 19005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:37:31.426633 19005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "7b89196309ef49deb820650c2461ff2e"
format_stamp: "Formatted at 2026-09-27 02:37:31 on dist-test-slave-grlk"
server_key: "bbe94596336dfed25781078e8a7f6a3c"
server_key_iv: "706b8f2b3744c99638e73465e1fca716"
server_key_version: "encryptionkey@0"
I20260927 02:37:31.427155 19005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "7b89196309ef49deb820650c2461ff2e"
format_stamp: "Formatted at 2026-09-27 02:37:31 on dist-test-slave-grlk"
server_key: "bbe94596336dfed25781078e8a7f6a3c"
server_key_iv: "706b8f2b3744c99638e73465e1fca716"
server_key_version: "encryptionkey@0"
I20260927 02:37:31.430912 19005 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 02:37:31.433638 19019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:31.434913 19005 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:37:31.435041 19005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "7b89196309ef49deb820650c2461ff2e"
format_stamp: "Formatted at 2026-09-27 02:37:31 on dist-test-slave-grlk"
server_key: "bbe94596336dfed25781078e8a7f6a3c"
server_key_iv: "706b8f2b3744c99638e73465e1fca716"
server_key_version: "encryptionkey@0"
I20260927 02:37:31.435158 19005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:37:31.451715 19005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:37:31.452399 19005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:37:31.452561 19005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:37:31.459018 19005 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45259
I20260927 02:37:31.459038 19071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45259 every 8 connection(s)
I20260927 02:37:31.459899 19005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:37:31.462620 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 19005
I20260927 02:37:31.462711 19072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:37:31.462754 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:37:31.463024 392 external_mini_cluster.cc:1428] Setting key 91c36fbc1947d4f87dab2da4a0554016
I20260927 02:37:31.468597 19072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b89196309ef49deb820650c2461ff2e: Bootstrap starting.
I20260927 02:37:31.470742 19072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b89196309ef49deb820650c2461ff2e: Neither blocks nor log segments found. Creating new log.
I20260927 02:37:31.471470 19072 log.cc:824] T 00000000000000000000000000000000 P 7b89196309ef49deb820650c2461ff2e: Log is configured to *not* fsync() on all Append() calls
I20260927 02:37:31.471603 19075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:31.464334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37728 (local address 127.0.98.62:45259)
0927 02:37:31.464969 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:31.464987 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:31.465024 (+ 37us) server_negotiation.cc:407] Connection header received
0927 02:37:31.465714 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:31.465731 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:31.466104 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:31.466408 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:31.466791 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:31.467732 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:31.468570 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:31.469077 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:31.469218 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:31.470024 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:31.470028 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:31.470040 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:37:31.470402 (+ 362us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:31.470452 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:37:31.470471 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:31.470551 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:31.470678 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 02:37:31.470932 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":122,"threads_started":1}
I20260927 02:37:31.473613 19072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b89196309ef49deb820650c2461ff2e: No bootstrap required, opened a new log
I20260927 02:37:31.476859 19072 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b89196309ef49deb820650c2461ff2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b89196309ef49deb820650c2461ff2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45259 } }
I20260927 02:37:31.477075 19072 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b89196309ef49deb820650c2461ff2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:37:31.477128 19072 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b89196309ef49deb820650c2461ff2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b89196309ef49deb820650c2461ff2e, State: Initialized, Role: FOLLOWER
I20260927 02:37:31.477496 19072 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b89196309ef49deb820650c2461ff2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b89196309ef49deb820650c2461ff2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45259 } }
I20260927 02:37:31.477609 19072 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b89196309ef49deb820650c2461ff2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:37:31.477666 19072 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b89196309ef49deb820650c2461ff2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:37:31.477752 19072 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b89196309ef49deb820650c2461ff2e [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:37:31.478560 19072 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b89196309ef49deb820650c2461ff2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b89196309ef49deb820650c2461ff2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45259 } }
I20260927 02:37:31.478857 19072 leader_election.cc:302] T 00000000000000000000000000000000 P 7b89196309ef49deb820650c2461ff2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b89196309ef49deb820650c2461ff2e; no voters:
I20260927 02:37:31.479130 19072 leader_election.cc:288] T 00000000000000000000000000000000 P 7b89196309ef49deb820650c2461ff2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:37:31.479207 19077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b89196309ef49deb820650c2461ff2e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:37:31.479431 19077 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b89196309ef49deb820650c2461ff2e [term 1 LEADER]: Becoming Leader. State: Replica: 7b89196309ef49deb820650c2461ff2e, State: Running, Role: LEADER
I20260927 02:37:31.479744 19077 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b89196309ef49deb820650c2461ff2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b89196309ef49deb820650c2461ff2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45259 } }
I20260927 02:37:31.480268 19072 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b89196309ef49deb820650c2461ff2e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:37:31.481245 19078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b89196309ef49deb820650c2461ff2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b89196309ef49deb820650c2461ff2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b89196309ef49deb820650c2461ff2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45259 } } }
I20260927 02:37:31.481267 19079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b89196309ef49deb820650c2461ff2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b89196309ef49deb820650c2461ff2e. Latest consensus state: current_term: 1 leader_uuid: "7b89196309ef49deb820650c2461ff2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b89196309ef49deb820650c2461ff2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45259 } } }
I20260927 02:37:31.481359 19078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b89196309ef49deb820650c2461ff2e [sys.catalog]: This master's current role is: LEADER
I20260927 02:37:31.481366 19079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b89196309ef49deb820650c2461ff2e [sys.catalog]: This master's current role is: LEADER
I20260927 02:37:31.481678 19086 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:37:31.484433 19086 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:37:31.489311 19086 catalog_manager.cc:1347] Generated new cluster ID: beca3fa368be44d288e5aaf3b78354e7
I20260927 02:37:31.489357 19086 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:37:31.498463 19086 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:37:31.499238 19086 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:37:31.506430 19086 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b89196309ef49deb820650c2461ff2e: Generated new TSK 0
I20260927 02:37:31.506978 19086 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:37:31.511996 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45259
--builtin_ntp_servers=127.0.98.20:46559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 02:37:31.615098 19096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:37:31.615316 19096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:37:31.615365 19096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:37:31.619376 19096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:37:31.619424 19096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:37:31.619494 19096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:37:31.624212 19096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:37:31.625120 19096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:37:31.626336 19096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:37:31.634935 19096 server_base.cc:1034] running on GCE node
W20260927 02:37:31.634905 19102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:31.634965 19101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:31.634905 19104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:31.635416 19096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:37:31.635939 19096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:37:31.637082 19096 hybrid_clock.cc:648] HybridClock initialized: now 1790476651637068 us; error 24 us; skew 500 ppm
I20260927 02:37:31.638619 19096 webserver.cc:460] Webserver started at http://127.0.98.1:40527/ using document root <none> and password file <none>
I20260927 02:37:31.639137 19096 fs_manager.cc:359] Metadata directory not provided
I20260927 02:37:31.639194 19096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:37:31.639366 19096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:37:31.641186 19096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-0/data/instance:
uuid: "2c12274c2ef14f749d24aa8659a69290"
format_stamp: "Formatted at 2026-09-27 02:37:31 on dist-test-slave-grlk"
server_key: "c9244de9f6a15334809130927a6772fb"
server_key_iv: "f2c35cc783d3aa8e96d6c21da713c9ac"
server_key_version: "encryptionkey@0"
I20260927 02:37:31.641661 19096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2c12274c2ef14f749d24aa8659a69290"
format_stamp: "Formatted at 2026-09-27 02:37:31 on dist-test-slave-grlk"
server_key: "c9244de9f6a15334809130927a6772fb"
server_key_iv: "f2c35cc783d3aa8e96d6c21da713c9ac"
server_key_version: "encryptionkey@0"
I20260927 02:37:31.644806 19096 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 02:37:31.646879 19110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:31.647917 19096 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:37:31.648011 19096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-0/wal
uuid: "2c12274c2ef14f749d24aa8659a69290"
format_stamp: "Formatted at 2026-09-27 02:37:31 on dist-test-slave-grlk"
server_key: "c9244de9f6a15334809130927a6772fb"
server_key_iv: "f2c35cc783d3aa8e96d6c21da713c9ac"
server_key_version: "encryptionkey@0"
I20260927 02:37:31.648092 19096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:37:31.658541 19096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:37:31.659049 19096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:37:31.659186 19096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:37:31.659641 19096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:37:31.660413 19096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:37:31.660456 19096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:31.660503 19096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:37:31.660529 19096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:31.668128 19096 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45333
I20260927 02:37:31.668186 19223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45333 every 8 connection(s)
I20260927 02:37:31.668876 19096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:37:31.677042 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 19096
I20260927 02:37:31.677075 19075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:31.670425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60349 (local address 127.0.98.62:45259)
0927 02:37:31.670544 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:31.670548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:31.671499 (+ 951us) server_negotiation.cc:407] Connection header received
0927 02:37:31.672780 (+ 1281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:31.672783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:31.672837 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:31.672905 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:31.673776 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:31.674304 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:31.675375 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:31.675590 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:31.675683 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:31.676599 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:31.676603 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:31.676605 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:37:31.676616 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:31.676632 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:37:31.676647 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:31.676703 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:31.676848 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 02:37:31.676913 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 02:37:31.677155 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:37:31.677421 392 external_mini_cluster.cc:1428] Setting key e30e67c3dc8b791eaabb1ab8504d58d1
I20260927 02:37:31.677814 19226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:31.670733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45259 (local address 127.0.98.1:60349)
0927 02:37:31.671352 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:31.671389 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:31.672580 (+ 1191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:31.673138 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:31.673151 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:31.673227 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:37:31.673651 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:31.673662 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:31.674429 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:31.674432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:31.675265 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:31.675274 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:31.675720 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:31.675724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:31.675863 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:31.676372 (+ 509us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:37:31.676374 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:37:31.676409 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:37:31.676464 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:37:31.676713 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:37:31.676722 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:37:31.676727 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:37:31.676839 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 02:37:31.677099 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":99,"threads_started":1}
I20260927 02:37:31.679025 19224 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45259
I20260927 02:37:31.679178 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45259
--builtin_ntp_servers=127.0.98.20:46559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 02:37:31.679224 19224 heartbeater.cc:464] Registering TS with master...
I20260927 02:37:31.679663 19224 heartbeater.cc:511] Master 127.0.98.62:45259 requested a full tablet report, sending...
I20260927 02:37:31.681066 19036 ts_manager.cc:194] Registered new tserver with Master: 2c12274c2ef14f749d24aa8659a69290 (127.0.98.1:45333)
I20260927 02:37:31.682175 19036 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:60349
W20260927 02:37:31.781142 19227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:37:31.781359 19227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:37:31.781410 19227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:37:31.785331 19227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:37:31.785372 19227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:37:31.785442 19227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:37:31.789894 19227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:37:31.790733 19227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:37:31.791906 19227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:37:31.800710 19233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:31.800791 19235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:31.800810 19232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:31.801021 19227 server_base.cc:1034] running on GCE node
I20260927 02:37:31.801419 19227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:37:31.801966 19227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:37:31.803114 19227 hybrid_clock.cc:648] HybridClock initialized: now 1790476651803094 us; error 31 us; skew 500 ppm
I20260927 02:37:31.804715 19227 webserver.cc:460] Webserver started at http://127.0.98.2:35873/ using document root <none> and password file <none>
I20260927 02:37:31.805271 19227 fs_manager.cc:359] Metadata directory not provided
I20260927 02:37:31.805334 19227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:37:31.805498 19227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:37:31.807525 19227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-1/data/instance:
uuid: "7ff2e41f1a9b4b80993bdf4be9ed2fd6"
format_stamp: "Formatted at 2026-09-27 02:37:31 on dist-test-slave-grlk"
server_key: "b918255b81c54a329f706ee93dda6742"
server_key_iv: "14627858e6d79f1f0445525715cff13d"
server_key_version: "encryptionkey@0"
I20260927 02:37:31.808025 19227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7ff2e41f1a9b4b80993bdf4be9ed2fd6"
format_stamp: "Formatted at 2026-09-27 02:37:31 on dist-test-slave-grlk"
server_key: "b918255b81c54a329f706ee93dda6742"
server_key_iv: "14627858e6d79f1f0445525715cff13d"
server_key_version: "encryptionkey@0"
I20260927 02:37:31.811286 19227 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:37:31.813393 19241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:31.814458 19227 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:37:31.814563 19227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-1/wal
uuid: "7ff2e41f1a9b4b80993bdf4be9ed2fd6"
format_stamp: "Formatted at 2026-09-27 02:37:31 on dist-test-slave-grlk"
server_key: "b918255b81c54a329f706ee93dda6742"
server_key_iv: "14627858e6d79f1f0445525715cff13d"
server_key_version: "encryptionkey@0"
I20260927 02:37:31.814648 19227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:37:31.847831 19227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:37:31.848502 19227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:37:31.848662 19227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:37:31.849257 19227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:37:31.850138 19227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:37:31.850183 19227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:31.850234 19227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:37:31.850270 19227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:31.858381 19227 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35121
I20260927 02:37:31.858410 19354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35121 every 8 connection(s)
I20260927 02:37:31.859179 19227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:37:31.864408 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 19227
I20260927 02:37:31.864511 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:37:31.864769 392 external_mini_cluster.cc:1428] Setting key 93320f71abef6018b55a44c317f04d68
I20260927 02:37:31.866678 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45259
--builtin_ntp_servers=127.0.98.20:46559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 02:37:31.868664 19075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:31.860970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44657 (local address 127.0.98.62:45259)
0927 02:37:31.861101 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:31.861105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:31.862034 (+ 929us) server_negotiation.cc:407] Connection header received
0927 02:37:31.863324 (+ 1290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:31.863327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:31.863383 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:31.863454 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:31.864586 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:31.865327 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:31.866672 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:31.866977 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:31.867066 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:31.868190 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:31.868194 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:31.868197 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:37:31.868213 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:31.868239 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:37:31.868251 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:31.868333 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:31.868438 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 02:37:31.868505 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 02:37:31.869374 19357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:31.861310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45259 (local address 127.0.98.2:44657)
0927 02:37:31.861902 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:31.861939 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:31.863128 (+ 1189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:31.863605 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:31.863612 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:31.863694 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:37:31.864130 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:31.864142 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:31.865466 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:31.865469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:31.866546 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:31.866557 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:31.867117 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:31.867121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:31.867308 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:31.867910 (+ 602us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:37:31.867912 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:37:31.867965 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:37:31.868026 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:37:31.868329 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:37:31.868334 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:37:31.868337 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:37:31.868415 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 02:37:31.868610 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":106,"threads_started":1}
I20260927 02:37:31.870401 19355 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45259
I20260927 02:37:31.870635 19355 heartbeater.cc:464] Registering TS with master...
I20260927 02:37:31.871212 19355 heartbeater.cc:511] Master 127.0.98.62:45259 requested a full tablet report, sending...
I20260927 02:37:31.872279 19036 ts_manager.cc:194] Registered new tserver with Master: 7ff2e41f1a9b4b80993bdf4be9ed2fd6 (127.0.98.2:35121)
I20260927 02:37:31.872833 19036 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:44657
W20260927 02:37:31.972839 19358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:37:31.973073 19358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:37:31.973107 19358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:37:31.977022 19358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:37:31.977070 19358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:37:31.977123 19358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:37:31.981900 19358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:37:31.982802 19358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:37:31.984028 19358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:37:31.992839 19363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:31.992854 19364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:31.993045 19366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:31.992895 19358 server_base.cc:1034] running on GCE node
I20260927 02:37:31.993479 19358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:37:31.994066 19358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:37:31.995218 19358 hybrid_clock.cc:648] HybridClock initialized: now 1790476651995204 us; error 28 us; skew 500 ppm
I20260927 02:37:31.996845 19358 webserver.cc:460] Webserver started at http://127.0.98.3:40987/ using document root <none> and password file <none>
I20260927 02:37:31.997376 19358 fs_manager.cc:359] Metadata directory not provided
I20260927 02:37:31.997439 19358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:37:31.997608 19358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:37:31.999447 19358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-2/data/instance:
uuid: "439ced7f3a6d42e993e1b0ad1f1f8a51"
format_stamp: "Formatted at 2026-09-27 02:37:31 on dist-test-slave-grlk"
server_key: "86484b746e5dd73f30db6ec6c40312a4"
server_key_iv: "76e532fef998d779dbf7ede53cfba880"
server_key_version: "encryptionkey@0"
I20260927 02:37:31.999938 19358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-2/wal/instance:
uuid: "439ced7f3a6d42e993e1b0ad1f1f8a51"
format_stamp: "Formatted at 2026-09-27 02:37:31 on dist-test-slave-grlk"
server_key: "86484b746e5dd73f30db6ec6c40312a4"
server_key_iv: "76e532fef998d779dbf7ede53cfba880"
server_key_version: "encryptionkey@0"
I20260927 02:37:32.003243 19358 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:37:32.005432 19372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:32.006590 19358 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:37:32.006690 19358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-2/wal
uuid: "439ced7f3a6d42e993e1b0ad1f1f8a51"
format_stamp: "Formatted at 2026-09-27 02:37:31 on dist-test-slave-grlk"
server_key: "86484b746e5dd73f30db6ec6c40312a4"
server_key_iv: "76e532fef998d779dbf7ede53cfba880"
server_key_version: "encryptionkey@0"
I20260927 02:37:32.006775 19358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:37:32.018373 19358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:37:32.018929 19358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:37:32.019074 19358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:37:32.019585 19358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:37:32.020427 19358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:37:32.020473 19358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:32.020521 19358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:37:32.020547 19358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:32.028381 19358 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35477
I20260927 02:37:32.028434 19485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35477 every 8 connection(s)
I20260927 02:37:32.029217 19358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:37:32.032506 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 19358
I20260927 02:37:32.032588 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:37:32.032809 392 external_mini_cluster.cc:1428] Setting key ac62615e4477fd151af144ecee29388e
I20260927 02:37:32.037523 19075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:32.030774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38599 (local address 127.0.98.62:45259)
0927 02:37:32.030894 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:32.030898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:32.031816 (+ 918us) server_negotiation.cc:407] Connection header received
0927 02:37:32.033205 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:32.033209 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:32.033275 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:32.033343 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:32.034398 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:32.034962 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:32.036002 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:32.036202 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:32.036267 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:32.037177 (+ 910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:32.037180 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:32.037182 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:37:32.037193 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:32.037212 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:37:32.037220 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:32.037268 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:32.037370 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 02:37:32.037416 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 02:37:32.038308 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:32.031109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45259 (local address 127.0.98.3:38599)
0927 02:37:32.031668 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:32.031704 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:32.033012 (+ 1308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:32.033634 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:32.033640 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:32.033727 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:37:32.034268 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:32.034285 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:32.035066 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:32.035068 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:32.035914 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:32.035923 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:32.036302 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:32.036305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:32.036432 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:32.036919 (+ 487us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:37:32.036920 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:37:32.036989 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:37:32.037047 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:37:32.037277 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:37:32.037281 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:37:32.037285 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:37:32.037365 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 02:37:32.037566 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":92,"threads_started":1}
I20260927 02:37:32.039275 19486 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45259
I20260927 02:37:32.039480 19486 heartbeater.cc:464] Registering TS with master...
I20260927 02:37:32.039978 19486 heartbeater.cc:511] Master 127.0.98.62:45259 requested a full tablet report, sending...
I20260927 02:37:32.040832 19036 ts_manager.cc:194] Registered new tserver with Master: 439ced7f3a6d42e993e1b0ad1f1f8a51 (127.0.98.3:35477)
I20260927 02:37:32.041396 19036 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:38599
I20260927 02:37:32.046000 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:37:32.050859 19075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:32.047732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37740 (local address 127.0.98.62:45259)
0927 02:37:32.047834 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:32.047840 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:32.047991 (+ 151us) server_negotiation.cc:407] Connection header received
0927 02:37:32.048073 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:32.048074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:32.048107 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:32.048164 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:32.048351 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:32.048804 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:32.049634 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:32.049835 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:32.049902 (+ 67us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:32.050516 (+ 614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:32.050518 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:32.050520 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:37:32.050530 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:32.050546 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:37:32.050553 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:32.050642 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:32.050719 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 02:37:32.050745 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 02:37:32.053251 19036 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37740:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:37:32.056022 19036 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:37:32.069686 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:32.063611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35477 (local address 127.0.0.1:45302)
0927 02:37:32.064124 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:32.064141 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:32.064598 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:32.065247 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:32.065252 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:32.065273 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:37:32.065395 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:32.065401 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:32.066750 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:32.066753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:32.068297 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:32.068302 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:32.069309 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:32.069311 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:32.069400 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:32.069429 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 02:37:32.069499 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 02:37:32.069576 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":147,"thread_start_us":88,"threads_started":1}
I20260927 02:37:32.069762 19501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:32.064205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45302 (local address 127.0.98.3:35477)
0927 02:37:32.064799 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:32.064809 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:32.064832 (+ 23us) server_negotiation.cc:407] Connection header received
0927 02:37:32.064903 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:32.064908 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:32.065102 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:32.065238 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:37:32.065492 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:32.066632 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:32.068395 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:32.069196 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:32.069307 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:32.069490 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:32.069545 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 02:37:32.069621 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":80,"threads_started":1}
I20260927 02:37:32.070267 19498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:32.063825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45333 (local address 127.0.0.1:52950)
0927 02:37:32.064763 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:32.064775 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:32.065589 (+ 814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:32.066043 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:32.066047 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:32.066063 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:37:32.066154 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:32.066160 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:32.067292 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:32.067296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:32.068238 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:32.068247 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:32.069907 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:32.069912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:32.069997 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:32.070021 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 02:37:32.070087 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 02:37:32.070148 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":878,"mutex_wait_us":19,"thread_start_us":95,"threads_started":1}
I20260927 02:37:32.070667 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:32.064206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52950 (local address 127.0.98.1:45333)
0927 02:37:32.064624 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:32.064635 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:32.064856 (+ 221us) server_negotiation.cc:407] Connection header received
0927 02:37:32.065778 (+ 922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:32.065782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:32.065918 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:32.066046 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:37:32.066261 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:32.067168 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:32.068358 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:32.069196 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:32.069309 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:32.069490 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:32.070426 (+ 936us) server_negotiation.cc:299] Negotiation successful
0927 02:37:32.070517 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":103,"threads_started":1}
I20260927 02:37:32.071396 19497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:32.063499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35121 (local address 127.0.0.1:42562)
0927 02:37:32.064081 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:32.064142 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:32.064270 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:32.065434 (+ 1164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:32.065437 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:32.065455 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:37:32.065566 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:32.065574 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:32.066847 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:32.066851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:32.068292 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:32.068302 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:32.070964 (+ 2662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:32.070968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:32.071062 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:32.071104 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 02:37:32.071187 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 02:37:32.071261 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"mutex_wait_us":93,"thread_start_us":57,"threads_started":1}
I20260927 02:37:32.071851 19499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:32.063634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42562 (local address 127.0.98.2:35121)
0927 02:37:32.065071 (+ 1437us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:32.065081 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:32.065100 (+ 19us) server_negotiation.cc:407] Connection header received
0927 02:37:32.065164 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:32.065167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:32.065304 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:32.065439 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:37:32.065693 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:32.066752 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:32.069937 (+ 3185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:32.070619 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:32.070725 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:32.070904 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:32.071569 (+ 665us) server_negotiation.cc:299] Negotiation successful
0927 02:37:32.071675 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1343,"thread_start_us":101,"threads_started":1}
I20260927 02:37:32.072747 19420 tablet_service.cc:1467] Processing CreateTablet for tablet 5a7a0c6a323e4fb9be6acd1519db0956 (DEFAULT_TABLE table=test-table [id=e5319697ccda4142a3dd306c3b18e210]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:37:32.073359 19158 tablet_service.cc:1467] Processing CreateTablet for tablet 5a7a0c6a323e4fb9be6acd1519db0956 (DEFAULT_TABLE table=test-table [id=e5319697ccda4142a3dd306c3b18e210]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:37:32.073873 19420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a7a0c6a323e4fb9be6acd1519db0956. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:37:32.074272 19158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a7a0c6a323e4fb9be6acd1519db0956. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:37:32.078384 19289 tablet_service.cc:1467] Processing CreateTablet for tablet 5a7a0c6a323e4fb9be6acd1519db0956 (DEFAULT_TABLE table=test-table [id=e5319697ccda4142a3dd306c3b18e210]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:37:32.079213 19289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a7a0c6a323e4fb9be6acd1519db0956. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:37:32.080976 19503 tablet_bootstrap.cc:492] T 5a7a0c6a323e4fb9be6acd1519db0956 P 2c12274c2ef14f749d24aa8659a69290: Bootstrap starting.
I20260927 02:37:32.082046 19504 tablet_bootstrap.cc:492] T 5a7a0c6a323e4fb9be6acd1519db0956 P 439ced7f3a6d42e993e1b0ad1f1f8a51: Bootstrap starting.
I20260927 02:37:32.083534 19503 tablet_bootstrap.cc:654] T 5a7a0c6a323e4fb9be6acd1519db0956 P 2c12274c2ef14f749d24aa8659a69290: Neither blocks nor log segments found. Creating new log.
I20260927 02:37:32.083706 19504 tablet_bootstrap.cc:654] T 5a7a0c6a323e4fb9be6acd1519db0956 P 439ced7f3a6d42e993e1b0ad1f1f8a51: Neither blocks nor log segments found. Creating new log.
I20260927 02:37:32.084314 19504 log.cc:824] T 5a7a0c6a323e4fb9be6acd1519db0956 P 439ced7f3a6d42e993e1b0ad1f1f8a51: Log is configured to *not* fsync() on all Append() calls
I20260927 02:37:32.084381 19503 log.cc:824] T 5a7a0c6a323e4fb9be6acd1519db0956 P 2c12274c2ef14f749d24aa8659a69290: Log is configured to *not* fsync() on all Append() calls
I20260927 02:37:32.085990 19507 tablet_bootstrap.cc:492] T 5a7a0c6a323e4fb9be6acd1519db0956 P 7ff2e41f1a9b4b80993bdf4be9ed2fd6: Bootstrap starting.
I20260927 02:37:32.086284 19504 tablet_bootstrap.cc:492] T 5a7a0c6a323e4fb9be6acd1519db0956 P 439ced7f3a6d42e993e1b0ad1f1f8a51: No bootstrap required, opened a new log
I20260927 02:37:32.086400 19503 tablet_bootstrap.cc:492] T 5a7a0c6a323e4fb9be6acd1519db0956 P 2c12274c2ef14f749d24aa8659a69290: No bootstrap required, opened a new log
I20260927 02:37:32.086495 19504 ts_tablet_manager.cc:1397] T 5a7a0c6a323e4fb9be6acd1519db0956 P 439ced7f3a6d42e993e1b0ad1f1f8a51: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 02:37:32.086576 19503 ts_tablet_manager.cc:1397] T 5a7a0c6a323e4fb9be6acd1519db0956 P 2c12274c2ef14f749d24aa8659a69290: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 02:37:32.088039 19507 tablet_bootstrap.cc:654] T 5a7a0c6a323e4fb9be6acd1519db0956 P 7ff2e41f1a9b4b80993bdf4be9ed2fd6: Neither blocks nor log segments found. Creating new log.
I20260927 02:37:32.088774 19507 log.cc:824] T 5a7a0c6a323e4fb9be6acd1519db0956 P 7ff2e41f1a9b4b80993bdf4be9ed2fd6: Log is configured to *not* fsync() on all Append() calls
I20260927 02:37:32.089841 19503 raft_consensus.cc:348] T 5a7a0c6a323e4fb9be6acd1519db0956 P 2c12274c2ef14f749d24aa8659a69290 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff2e41f1a9b4b80993bdf4be9ed2fd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35121 } } peers { permanent_uuid: "439ced7f3a6d42e993e1b0ad1f1f8a51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } } peers { permanent_uuid: "2c12274c2ef14f749d24aa8659a69290" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45333 } }
I20260927 02:37:32.090099 19503 raft_consensus.cc:374] T 5a7a0c6a323e4fb9be6acd1519db0956 P 2c12274c2ef14f749d24aa8659a69290 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:37:32.090163 19503 raft_consensus.cc:729] T 5a7a0c6a323e4fb9be6acd1519db0956 P 2c12274c2ef14f749d24aa8659a69290 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c12274c2ef14f749d24aa8659a69290, State: Initialized, Role: FOLLOWER
I20260927 02:37:32.090245 19504 raft_consensus.cc:348] T 5a7a0c6a323e4fb9be6acd1519db0956 P 439ced7f3a6d42e993e1b0ad1f1f8a51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff2e41f1a9b4b80993bdf4be9ed2fd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35121 } } peers { permanent_uuid: "439ced7f3a6d42e993e1b0ad1f1f8a51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } } peers { permanent_uuid: "2c12274c2ef14f749d24aa8659a69290" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45333 } }
I20260927 02:37:32.090477 19504 raft_consensus.cc:374] T 5a7a0c6a323e4fb9be6acd1519db0956 P 439ced7f3a6d42e993e1b0ad1f1f8a51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:37:32.090567 19504 raft_consensus.cc:729] T 5a7a0c6a323e4fb9be6acd1519db0956 P 439ced7f3a6d42e993e1b0ad1f1f8a51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 439ced7f3a6d42e993e1b0ad1f1f8a51, State: Initialized, Role: FOLLOWER
I20260927 02:37:32.090641 19507 tablet_bootstrap.cc:492] T 5a7a0c6a323e4fb9be6acd1519db0956 P 7ff2e41f1a9b4b80993bdf4be9ed2fd6: No bootstrap required, opened a new log
I20260927 02:37:32.090590 19503 consensus_queue.cc:260] T 5a7a0c6a323e4fb9be6acd1519db0956 P 2c12274c2ef14f749d24aa8659a69290 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff2e41f1a9b4b80993bdf4be9ed2fd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35121 } } peers { permanent_uuid: "439ced7f3a6d42e993e1b0ad1f1f8a51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } } peers { permanent_uuid: "2c12274c2ef14f749d24aa8659a69290" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45333 } }
I20260927 02:37:32.090845 19507 ts_tablet_manager.cc:1397] T 5a7a0c6a323e4fb9be6acd1519db0956 P 7ff2e41f1a9b4b80993bdf4be9ed2fd6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 02:37:32.090976 19504 consensus_queue.cc:260] T 5a7a0c6a323e4fb9be6acd1519db0956 P 439ced7f3a6d42e993e1b0ad1f1f8a51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff2e41f1a9b4b80993bdf4be9ed2fd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35121 } } peers { permanent_uuid: "439ced7f3a6d42e993e1b0ad1f1f8a51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } } peers { permanent_uuid: "2c12274c2ef14f749d24aa8659a69290" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45333 } }
I20260927 02:37:32.091266 19224 heartbeater.cc:503] Master 127.0.98.62:45259 was elected leader, sending a full tablet report...
I20260927 02:37:32.091540 19503 ts_tablet_manager.cc:1428] T 5a7a0c6a323e4fb9be6acd1519db0956 P 2c12274c2ef14f749d24aa8659a69290: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 02:37:32.092027 19504 ts_tablet_manager.cc:1428] T 5a7a0c6a323e4fb9be6acd1519db0956 P 439ced7f3a6d42e993e1b0ad1f1f8a51: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260927 02:37:32.092181 19486 heartbeater.cc:503] Master 127.0.98.62:45259 was elected leader, sending a full tablet report...
I20260927 02:37:32.095374 19507 raft_consensus.cc:348] T 5a7a0c6a323e4fb9be6acd1519db0956 P 7ff2e41f1a9b4b80993bdf4be9ed2fd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff2e41f1a9b4b80993bdf4be9ed2fd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35121 } } peers { permanent_uuid: "439ced7f3a6d42e993e1b0ad1f1f8a51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } } peers { permanent_uuid: "2c12274c2ef14f749d24aa8659a69290" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45333 } }
I20260927 02:37:32.095640 19507 raft_consensus.cc:374] T 5a7a0c6a323e4fb9be6acd1519db0956 P 7ff2e41f1a9b4b80993bdf4be9ed2fd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:37:32.095696 19507 raft_consensus.cc:729] T 5a7a0c6a323e4fb9be6acd1519db0956 P 7ff2e41f1a9b4b80993bdf4be9ed2fd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ff2e41f1a9b4b80993bdf4be9ed2fd6, State: Initialized, Role: FOLLOWER
I20260927 02:37:32.096029 19507 consensus_queue.cc:260] T 5a7a0c6a323e4fb9be6acd1519db0956 P 7ff2e41f1a9b4b80993bdf4be9ed2fd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff2e41f1a9b4b80993bdf4be9ed2fd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35121 } } peers { permanent_uuid: "439ced7f3a6d42e993e1b0ad1f1f8a51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } } peers { permanent_uuid: "2c12274c2ef14f749d24aa8659a69290" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45333 } }
I20260927 02:37:32.096563 19355 heartbeater.cc:503] Master 127.0.98.62:45259 was elected leader, sending a full tablet report...
I20260927 02:37:32.096745 19507 ts_tablet_manager.cc:1428] T 5a7a0c6a323e4fb9be6acd1519db0956 P 7ff2e41f1a9b4b80993bdf4be9ed2fd6: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260927 02:37:32.109953 19356 tablet.cc:2369] T 5a7a0c6a323e4fb9be6acd1519db0956 P 7ff2e41f1a9b4b80993bdf4be9ed2fd6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:37:32.163388 19509 raft_consensus.cc:482] T 5a7a0c6a323e4fb9be6acd1519db0956 P 2c12274c2ef14f749d24aa8659a69290 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:37:32.163585 19509 raft_consensus.cc:504] T 5a7a0c6a323e4fb9be6acd1519db0956 P 2c12274c2ef14f749d24aa8659a69290 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff2e41f1a9b4b80993bdf4be9ed2fd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35121 } } peers { permanent_uuid: "439ced7f3a6d42e993e1b0ad1f1f8a51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } } peers { permanent_uuid: "2c12274c2ef14f749d24aa8659a69290" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45333 } }
I20260927 02:37:32.164716 19509 leader_election.cc:288] T 5a7a0c6a323e4fb9be6acd1519db0956 P 2c12274c2ef14f749d24aa8659a69290 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ff2e41f1a9b4b80993bdf4be9ed2fd6 (127.0.98.2:35121), 439ced7f3a6d42e993e1b0ad1f1f8a51 (127.0.98.3:35477)
I20260927 02:37:32.168035 19226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:32.164891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35121 (local address 127.0.98.1:53877)
0927 02:37:32.165090 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:32.165104 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:32.165187 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:32.165447 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:32.165450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:32.165468 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:37:32.165570 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:32.165576 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:32.166275 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:32.166279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:32.167035 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:32.167044 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:32.167645 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:32.167649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:32.167743 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:32.167765 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 02:37:32.167837 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 02:37:32.167906 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":1}
I20260927 02:37:32.168035 19499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:32.165031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53877 (local address 127.0.98.2:35121)
0927 02:37:32.165142 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:32.165146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:32.165161 (+ 15us) server_negotiation.cc:407] Connection header received
0927 02:37:32.165291 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:32.165294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:32.165348 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:32.165416 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:37:32.165675 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:32.166174 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:32.167154 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:32.167539 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:32.167627 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:32.167686 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:32.167859 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 02:37:32.167927 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 02:37:32.168278 19512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:32.164962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35477 (local address 127.0.98.1:38793)
0927 02:37:32.165332 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:32.165343 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:32.165425 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:32.165646 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:32.165649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:32.165664 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:37:32.165787 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:32.165791 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:32.166519 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:32.166523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:32.167290 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:32.167298 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:32.167899 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:32.167902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:32.167992 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:32.168014 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 02:37:32.168075 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 02:37:32.168139 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":81,"threads_started":1}
I20260927 02:37:32.168272 19501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:32.165099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38793 (local address 127.0.98.3:35477)
0927 02:37:32.165214 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:32.165219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:32.165411 (+ 192us) server_negotiation.cc:407] Connection header received
0927 02:37:32.165509 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:32.165512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:32.165554 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:32.165614 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:37:32.165883 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:32.166425 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:32.167402 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:32.167788 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:32.167875 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:32.167931 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:32.168096 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 02:37:32.168162 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 02:37:32.168907 19309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a7a0c6a323e4fb9be6acd1519db0956" candidate_uuid: "2c12274c2ef14f749d24aa8659a69290" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ff2e41f1a9b4b80993bdf4be9ed2fd6" is_pre_election: true
I20260927 02:37:32.168908 19440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a7a0c6a323e4fb9be6acd1519db0956" candidate_uuid: "2c12274c2ef14f749d24aa8659a69290" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "439ced7f3a6d42e993e1b0ad1f1f8a51" is_pre_election: true
I20260927 02:37:32.169214 19440 raft_consensus.cc:2454] T 5a7a0c6a323e4fb9be6acd1519db0956 P 439ced7f3a6d42e993e1b0ad1f1f8a51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c12274c2ef14f749d24aa8659a69290 in term 0.
I20260927 02:37:32.169214 19309 raft_consensus.cc:2454] T 5a7a0c6a323e4fb9be6acd1519db0956 P 7ff2e41f1a9b4b80993bdf4be9ed2fd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c12274c2ef14f749d24aa8659a69290 in term 0.
I20260927 02:37:32.169643 19114 leader_election.cc:302] T 5a7a0c6a323e4fb9be6acd1519db0956 P 2c12274c2ef14f749d24aa8659a69290 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c12274c2ef14f749d24aa8659a69290, 7ff2e41f1a9b4b80993bdf4be9ed2fd6; no voters:
W20260927 02:37:32.169843 19225 tablet.cc:2369] T 5a7a0c6a323e4fb9be6acd1519db0956 P 2c12274c2ef14f749d24aa8659a69290: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:37:32.169879 19509 raft_consensus.cc:2781] T 5a7a0c6a323e4fb9be6acd1519db0956 P 2c12274c2ef14f749d24aa8659a69290 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:37:32.169953 19509 raft_consensus.cc:482] T 5a7a0c6a323e4fb9be6acd1519db0956 P 2c12274c2ef14f749d24aa8659a69290 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:37:32.169992 19509 raft_consensus.cc:3037] T 5a7a0c6a323e4fb9be6acd1519db0956 P 2c12274c2ef14f749d24aa8659a69290 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:37:32.170917 19509 raft_consensus.cc:504] T 5a7a0c6a323e4fb9be6acd1519db0956 P 2c12274c2ef14f749d24aa8659a69290 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff2e41f1a9b4b80993bdf4be9ed2fd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35121 } } peers { permanent_uuid: "439ced7f3a6d42e993e1b0ad1f1f8a51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } } peers { permanent_uuid: "2c12274c2ef14f749d24aa8659a69290" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45333 } }
I20260927 02:37:32.171200 19509 leader_election.cc:288] T 5a7a0c6a323e4fb9be6acd1519db0956 P 2c12274c2ef14f749d24aa8659a69290 [CANDIDATE]: Term 1 election: Requested vote from peers 7ff2e41f1a9b4b80993bdf4be9ed2fd6 (127.0.98.2:35121), 439ced7f3a6d42e993e1b0ad1f1f8a51 (127.0.98.3:35477)
I20260927 02:37:32.171491 19309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a7a0c6a323e4fb9be6acd1519db0956" candidate_uuid: "2c12274c2ef14f749d24aa8659a69290" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ff2e41f1a9b4b80993bdf4be9ed2fd6"
I20260927 02:37:32.171512 19440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a7a0c6a323e4fb9be6acd1519db0956" candidate_uuid: "2c12274c2ef14f749d24aa8659a69290" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "439ced7f3a6d42e993e1b0ad1f1f8a51"
I20260927 02:37:32.171605 19309 raft_consensus.cc:3037] T 5a7a0c6a323e4fb9be6acd1519db0956 P 7ff2e41f1a9b4b80993bdf4be9ed2fd6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:37:32.171605 19440 raft_consensus.cc:3037] T 5a7a0c6a323e4fb9be6acd1519db0956 P 439ced7f3a6d42e993e1b0ad1f1f8a51 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:37:32.172719 19440 raft_consensus.cc:2454] T 5a7a0c6a323e4fb9be6acd1519db0956 P 439ced7f3a6d42e993e1b0ad1f1f8a51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c12274c2ef14f749d24aa8659a69290 in term 1.
I20260927 02:37:32.172719 19309 raft_consensus.cc:2454] T 5a7a0c6a323e4fb9be6acd1519db0956 P 7ff2e41f1a9b4b80993bdf4be9ed2fd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c12274c2ef14f749d24aa8659a69290 in term 1.
I20260927 02:37:32.173029 19113 leader_election.cc:302] T 5a7a0c6a323e4fb9be6acd1519db0956 P 2c12274c2ef14f749d24aa8659a69290 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2c12274c2ef14f749d24aa8659a69290, 439ced7f3a6d42e993e1b0ad1f1f8a51, 7ff2e41f1a9b4b80993bdf4be9ed2fd6; no voters:
I20260927 02:37:32.173221 19509 raft_consensus.cc:2781] T 5a7a0c6a323e4fb9be6acd1519db0956 P 2c12274c2ef14f749d24aa8659a69290 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:37:32.173388 19509 raft_consensus.cc:686] T 5a7a0c6a323e4fb9be6acd1519db0956 P 2c12274c2ef14f749d24aa8659a69290 [term 1 LEADER]: Becoming Leader. State: Replica: 2c12274c2ef14f749d24aa8659a69290, State: Running, Role: LEADER
I20260927 02:37:32.173655 19509 consensus_queue.cc:237] T 5a7a0c6a323e4fb9be6acd1519db0956 P 2c12274c2ef14f749d24aa8659a69290 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff2e41f1a9b4b80993bdf4be9ed2fd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35121 } } peers { permanent_uuid: "439ced7f3a6d42e993e1b0ad1f1f8a51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } } peers { permanent_uuid: "2c12274c2ef14f749d24aa8659a69290" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45333 } }
I20260927 02:37:32.176510 19035 catalog_manager.cc:5495] T 5a7a0c6a323e4fb9be6acd1519db0956 P 2c12274c2ef14f749d24aa8659a69290 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c12274c2ef14f749d24aa8659a69290 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "2c12274c2ef14f749d24aa8659a69290" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff2e41f1a9b4b80993bdf4be9ed2fd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "439ced7f3a6d42e993e1b0ad1f1f8a51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c12274c2ef14f749d24aa8659a69290" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45333 } health_report { overall_health: HEALTHY } } }
I20260927 02:37:32.187668 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:32.184856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52954 (local address 127.0.98.1:45333)
0927 02:37:32.184975 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:32.184979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:32.184992 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:37:32.185081 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:32.185083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:32.185127 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:32.185197 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:37:32.185383 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:32.185833 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:32.186571 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:32.186759 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:32.186821 (+ 62us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:32.187041 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:37:32.187383 (+ 342us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:37:32.187480 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:32.187538 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 02:37:32.187564 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 02:37:32.192456 19440 raft_consensus.cc:1261] T 5a7a0c6a323e4fb9be6acd1519db0956 P 439ced7f3a6d42e993e1b0ad1f1f8a51 [term 1 FOLLOWER]: Refusing update from remote peer 2c12274c2ef14f749d24aa8659a69290: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:37:32.192456 19309 raft_consensus.cc:1261] T 5a7a0c6a323e4fb9be6acd1519db0956 P 7ff2e41f1a9b4b80993bdf4be9ed2fd6 [term 1 FOLLOWER]: Refusing update from remote peer 2c12274c2ef14f749d24aa8659a69290: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:37:32.193023 19513 consensus_queue.cc:1035] T 5a7a0c6a323e4fb9be6acd1519db0956 P 2c12274c2ef14f749d24aa8659a69290 [LEADER]: Connected to new peer: Peer: permanent_uuid: "439ced7f3a6d42e993e1b0ad1f1f8a51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:37:32.193185 19509 consensus_queue.cc:1035] T 5a7a0c6a323e4fb9be6acd1519db0956 P 2c12274c2ef14f749d24aa8659a69290 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ff2e41f1a9b4b80993bdf4be9ed2fd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:37:32.200371 19517 mvcc.cc:205] Tried to move back new op lower bound from 7333792367374741504 to 7333792367305560064. Current Snapshot: MvccSnapshot[applied={T|T < 7333792367374741504}]
I20260927 02:37:32.201095 19518 mvcc.cc:205] Tried to move back new op lower bound from 7333792367374741504 to 7333792367305560064. Current Snapshot: MvccSnapshot[applied={T|T < 7333792367374741504}]
I20260927 02:37:32.207016 19036 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:37:32.207137 19036 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:37:32.209223 19036 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7b89196309ef49deb820650c2461ff2e: Sending DeleteTablet for 3 replicas of tablet 5a7a0c6a323e4fb9be6acd1519db0956
I20260927 02:37:32.209830 19420 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a7a0c6a323e4fb9be6acd1519db0956 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:37:32 UTC) from {username='slave'} at 127.0.0.1:45302
I20260927 02:37:32.209858 19289 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a7a0c6a323e4fb9be6acd1519db0956 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:37:32 UTC) from {username='slave'} at 127.0.0.1:42562
I20260927 02:37:32.209940 19158 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a7a0c6a323e4fb9be6acd1519db0956 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:37:32 UTC) from {username='slave'} at 127.0.0.1:52950
I20260927 02:37:32.210321 19527 tablet_replica.cc:330] T 5a7a0c6a323e4fb9be6acd1519db0956 P 439ced7f3a6d42e993e1b0ad1f1f8a51: stopping tablet replica
I20260927 02:37:32.210475 19528 tablet_replica.cc:330] T 5a7a0c6a323e4fb9be6acd1519db0956 P 2c12274c2ef14f749d24aa8659a69290: stopping tablet replica
I20260927 02:37:32.210590 19527 raft_consensus.cc:2229] T 5a7a0c6a323e4fb9be6acd1519db0956 P 439ced7f3a6d42e993e1b0ad1f1f8a51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:37:32.210654 19526 tablet_replica.cc:330] T 5a7a0c6a323e4fb9be6acd1519db0956 P 7ff2e41f1a9b4b80993bdf4be9ed2fd6: stopping tablet replica
I20260927 02:37:32.210870 19527 raft_consensus.cc:2258] T 5a7a0c6a323e4fb9be6acd1519db0956 P 439ced7f3a6d42e993e1b0ad1f1f8a51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:37:32.210939 19526 raft_consensus.cc:2229] T 5a7a0c6a323e4fb9be6acd1519db0956 P 7ff2e41f1a9b4b80993bdf4be9ed2fd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:37:32.211134 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 19096
I20260927 02:37:32.212189 19526 raft_consensus.cc:2258] T 5a7a0c6a323e4fb9be6acd1519db0956 P 7ff2e41f1a9b4b80993bdf4be9ed2fd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:37:32.212248 19527 ts_tablet_manager.cc:1905] T 5a7a0c6a323e4fb9be6acd1519db0956 P 439ced7f3a6d42e993e1b0ad1f1f8a51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:37:32.213486 19526 ts_tablet_manager.cc:1905] T 5a7a0c6a323e4fb9be6acd1519db0956 P 7ff2e41f1a9b4b80993bdf4be9ed2fd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:37:32.214725 19527 ts_tablet_manager.cc:1918] T 5a7a0c6a323e4fb9be6acd1519db0956 P 439ced7f3a6d42e993e1b0ad1f1f8a51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:37:32.214815 19527 log.cc:1196] T 5a7a0c6a323e4fb9be6acd1519db0956 P 439ced7f3a6d42e993e1b0ad1f1f8a51: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-2/wal/wals/5a7a0c6a323e4fb9be6acd1519db0956
I20260927 02:37:32.215101 19527 ts_tablet_manager.cc:1939] T 5a7a0c6a323e4fb9be6acd1519db0956 P 439ced7f3a6d42e993e1b0ad1f1f8a51: Deleting consensus metadata
I20260927 02:37:32.215919 19526 ts_tablet_manager.cc:1918] T 5a7a0c6a323e4fb9be6acd1519db0956 P 7ff2e41f1a9b4b80993bdf4be9ed2fd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:37:32.216006 19526 log.cc:1196] T 5a7a0c6a323e4fb9be6acd1519db0956 P 7ff2e41f1a9b4b80993bdf4be9ed2fd6: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476410933508-392-0/minicluster-data/ts-1/wal/wals/5a7a0c6a323e4fb9be6acd1519db0956
I20260927 02:37:32.216152 19022 catalog_manager.cc:4841] TS 439ced7f3a6d42e993e1b0ad1f1f8a51 (127.0.98.3:35477): tablet 5a7a0c6a323e4fb9be6acd1519db0956 (table test-table [id=e5319697ccda4142a3dd306c3b18e210]) successfully deleted
I20260927 02:37:32.216271 19526 ts_tablet_manager.cc:1939] T 5a7a0c6a323e4fb9be6acd1519db0956 P 7ff2e41f1a9b4b80993bdf4be9ed2fd6: Deleting consensus metadata
W20260927 02:37:32.217115 19020 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45333 (error 108)
I20260927 02:37:32.217592 19023 catalog_manager.cc:4841] TS 7ff2e41f1a9b4b80993bdf4be9ed2fd6 (127.0.98.2:35121): tablet 5a7a0c6a323e4fb9be6acd1519db0956 (table test-table [id=e5319697ccda4142a3dd306c3b18e210]) successfully deleted
I20260927 02:37:32.217873 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 19227
I20260927 02:37:32.217973 19498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:32.217721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45333 (local address 127.0.0.1:52960)
0927 02:37:32.217832 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:32.217900 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 02:37:32.218603 19020 catalog_manager.cc:4584] TS 2c12274c2ef14f749d24aa8659a69290 (127.0.98.1:45333): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5a7a0c6a323e4fb9be6acd1519db0956: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45333: connect: Connection refused (error 111)
I20260927 02:37:32.222646 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 19358
I20260927 02:37:32.227240 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 19005
2026-09-27T02:37:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (955 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:37:32 dist-test-slave-grlk krb5kdc[19532](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:37:32 dist-test-slave-grlk krb5kdc[19532](info): set up 2 sockets
Sep 27 02:37:32 dist-test-slave-grlk krb5kdc[19532](info): commencing operation
krb5kdc: starting...
W20260927 02:37:33.180603 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/krb5kdc/test-user.keytab.
Sep 27 02:37:33 dist-test-slave-grlk krb5kdc[19532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:37:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:37:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:37:33.285790 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35123
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35123
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:37:33.392180 19545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:37:33.392416 19545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:37:33.392467 19545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:37:33.396404 19545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:37:33.396454 19545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:37:33.396481 19545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:37:33.396502 19545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:37:33.396529 19545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:37:33.401571 19545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:37:33.402479 19545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:37:33.403389 19545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:37:33.411000 19551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:33.410995 19553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:33.411015 19550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:33.411216 19545 server_base.cc:1034] running on GCE node
I20260927 02:37:33.411768 19545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:37:33.412746 19545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:37:33.413899 19545 hybrid_clock.cc:648] HybridClock initialized: now 1790476653413879 us; error 33 us; skew 500 ppm
Sep 27 02:37:33 dist-test-slave-grlk krb5kdc[19532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:37:33.416416 19545 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:37:33.417253 19545 webserver.cc:460] Webserver started at http://127.0.98.62:39849/ using document root <none> and password file <none>
I20260927 02:37:33.417725 19545 fs_manager.cc:359] Metadata directory not provided
I20260927 02:37:33.417786 19545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:37:33.417950 19545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:37:33.419750 19545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "fd469a06e1f647d5bd336e888887529f"
format_stamp: "Formatted at 2026-09-27 02:37:33 on dist-test-slave-grlk"
server_key: "80b5942adbc615c51b5c8e6ac932dee4"
server_key_iv: "77d20f8fe7eb2a93c28c487efe3af9c7"
server_key_version: "encryptionkey@0"
I20260927 02:37:33.420246 19545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "fd469a06e1f647d5bd336e888887529f"
format_stamp: "Formatted at 2026-09-27 02:37:33 on dist-test-slave-grlk"
server_key: "80b5942adbc615c51b5c8e6ac932dee4"
server_key_iv: "77d20f8fe7eb2a93c28c487efe3af9c7"
server_key_version: "encryptionkey@0"
I20260927 02:37:33.423444 19545 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260927 02:37:33.425527 19560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:33.426582 19545 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 02:37:33.426685 19545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "fd469a06e1f647d5bd336e888887529f"
format_stamp: "Formatted at 2026-09-27 02:37:33 on dist-test-slave-grlk"
server_key: "80b5942adbc615c51b5c8e6ac932dee4"
server_key_iv: "77d20f8fe7eb2a93c28c487efe3af9c7"
server_key_version: "encryptionkey@0"
I20260927 02:37:33.426770 19545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:37:33.451942 19545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:37:33.454705 19545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:37:33.454861 19545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:37:33.461087 19545 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35123
I20260927 02:37:33.461127 19612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35123 every 8 connection(s)
I20260927 02:37:33.461930 19545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:37:33.464347 19613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:37:33.469273 19613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd469a06e1f647d5bd336e888887529f: Bootstrap starting.
I20260927 02:37:33.470852 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 19545
I20260927 02:37:33.470973 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:37:33.471216 392 external_mini_cluster.cc:1428] Setting key aa9fbe00f1ec3fef3176a440e318f4ce
I20260927 02:37:33.471534 19613 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd469a06e1f647d5bd336e888887529f: Neither blocks nor log segments found. Creating new log.
I20260927 02:37:33.472316 19613 log.cc:824] T 00000000000000000000000000000000 P fd469a06e1f647d5bd336e888887529f: Log is configured to *not* fsync() on all Append() calls
I20260927 02:37:33.474037 19613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd469a06e1f647d5bd336e888887529f: No bootstrap required, opened a new log
I20260927 02:37:33.477761 19613 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd469a06e1f647d5bd336e888887529f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd469a06e1f647d5bd336e888887529f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35123 } }
I20260927 02:37:33.477995 19613 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd469a06e1f647d5bd336e888887529f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:37:33.478053 19613 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd469a06e1f647d5bd336e888887529f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd469a06e1f647d5bd336e888887529f, State: Initialized, Role: FOLLOWER
I20260927 02:37:33.478506 19613 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd469a06e1f647d5bd336e888887529f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd469a06e1f647d5bd336e888887529f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35123 } }
I20260927 02:37:33.478653 19613 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd469a06e1f647d5bd336e888887529f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:37:33.478714 19613 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd469a06e1f647d5bd336e888887529f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:37:33.478806 19613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd469a06e1f647d5bd336e888887529f [term 0 FOLLOWER]: Advancing to term 1
Sep 27 02:37:33 dist-test-slave-grlk krb5kdc[19532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:37:33.479614 19613 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd469a06e1f647d5bd336e888887529f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd469a06e1f647d5bd336e888887529f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35123 } }
I20260927 02:37:33.479936 19613 leader_election.cc:302] T 00000000000000000000000000000000 P fd469a06e1f647d5bd336e888887529f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd469a06e1f647d5bd336e888887529f; no voters:
I20260927 02:37:33.480223 19613 leader_election.cc:288] T 00000000000000000000000000000000 P fd469a06e1f647d5bd336e888887529f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:37:33.480352 19618 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd469a06e1f647d5bd336e888887529f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:37:33.480554 19618 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd469a06e1f647d5bd336e888887529f [term 1 LEADER]: Becoming Leader. State: Replica: fd469a06e1f647d5bd336e888887529f, State: Running, Role: LEADER
I20260927 02:37:33.480880 19618 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd469a06e1f647d5bd336e888887529f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd469a06e1f647d5bd336e888887529f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35123 } }
I20260927 02:37:33.481467 19613 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd469a06e1f647d5bd336e888887529f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:37:33.482375 19619 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd469a06e1f647d5bd336e888887529f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd469a06e1f647d5bd336e888887529f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd469a06e1f647d5bd336e888887529f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35123 } } }
I20260927 02:37:33.482498 19619 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd469a06e1f647d5bd336e888887529f [sys.catalog]: This master's current role is: LEADER
I20260927 02:37:33.482831 19620 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd469a06e1f647d5bd336e888887529f [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd469a06e1f647d5bd336e888887529f. Latest consensus state: current_term: 1 leader_uuid: "fd469a06e1f647d5bd336e888887529f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd469a06e1f647d5bd336e888887529f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35123 } } }
I20260927 02:37:33.483530 19620 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd469a06e1f647d5bd336e888887529f [sys.catalog]: This master's current role is: LEADER
I20260927 02:37:33.483973 19627 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:37:33.484596 19616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:33.472153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60364 (local address 127.0.98.62:35123)
0927 02:37:33.472549 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:33.472563 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:33.472601 (+ 38us) server_negotiation.cc:407] Connection header received
0927 02:37:33.473323 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:33.473341 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:33.473630 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:33.473889 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:33.474726 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:33.475655 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:33.476822 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:33.477300 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:33.477409 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:33.479843 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:33.479848 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:33.479863 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:37:33.479891 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:33.481810 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:33.482274 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:33.482280 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:33.482285 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:33.482417 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:33.482715 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:33.482718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:33.482720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:33.483175 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:33.483360 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:33.483611 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 02:37:33.483895 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":124,"threads_started":1}
I20260927 02:37:33.486541 19627 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:37:33.491287 19627 catalog_manager.cc:1347] Generated new cluster ID: 28407bb3dc8f4389903330fc16cbbdc5
I20260927 02:37:33.491348 19627 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:37:33.515300 19627 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:37:33.516134 19627 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:37:33.521772 19627 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd469a06e1f647d5bd336e888887529f: Generated new TSK 0
I20260927 02:37:33.522305 19627 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:37:33.547674 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35123
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:37:33.648538 19639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:37:33.648748 19639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:37:33.648801 19639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:37:33.652618 19639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:37:33.652667 19639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:37:33.652743 19639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:37:33.657629 19639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:37:33.658468 19639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:37:33.659343 19639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:37:33.667989 19645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:33.667996 19644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:33.668110 19639 server_base.cc:1034] running on GCE node
W20260927 02:37:33.667996 19647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:33.668591 19639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:37:33.669201 19639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:37:33.670348 19639 hybrid_clock.cc:648] HybridClock initialized: now 1790476653670333 us; error 25 us; skew 500 ppm
Sep 27 02:37:33 dist-test-slave-grlk krb5kdc[19532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:37:33.672711 19639 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:37:33.673486 19639 webserver.cc:460] Webserver started at http://127.0.98.1:42037/ using document root <none> and password file <none>
I20260927 02:37:33.673972 19639 fs_manager.cc:359] Metadata directory not provided
I20260927 02:37:33.674032 19639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:37:33.674194 19639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:37:33.676007 19639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-0/data/instance:
uuid: "eaf3a85f105b42e0b6bc5629ccacfc7e"
format_stamp: "Formatted at 2026-09-27 02:37:33 on dist-test-slave-grlk"
server_key: "1c8f85b757c77ee5af585ee95594cb50"
server_key_iv: "56d5c5e8472f60edb57c7047c8a26327"
server_key_version: "encryptionkey@0"
I20260927 02:37:33.676537 19639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-0/wal/instance:
uuid: "eaf3a85f105b42e0b6bc5629ccacfc7e"
format_stamp: "Formatted at 2026-09-27 02:37:33 on dist-test-slave-grlk"
server_key: "1c8f85b757c77ee5af585ee95594cb50"
server_key_iv: "56d5c5e8472f60edb57c7047c8a26327"
server_key_version: "encryptionkey@0"
I20260927 02:37:33.679728 19639 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 02:37:33.681819 19654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:33.682973 19639 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:37:33.683068 19639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-0/wal
uuid: "eaf3a85f105b42e0b6bc5629ccacfc7e"
format_stamp: "Formatted at 2026-09-27 02:37:33 on dist-test-slave-grlk"
server_key: "1c8f85b757c77ee5af585ee95594cb50"
server_key_iv: "56d5c5e8472f60edb57c7047c8a26327"
server_key_version: "encryptionkey@0"
I20260927 02:37:33.683162 19639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:37:33.694821 19639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:37:33.697100 19639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:37:33.697252 19639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:37:33.697721 19639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:37:33.698418 19639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:37:33.698463 19639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:33.698508 19639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:37:33.698536 19639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:33.706295 19639 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34129
I20260927 02:37:33.706360 19767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34129 every 8 connection(s)
I20260927 02:37:33.707088 19639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:37:33.712522 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 19639
I20260927 02:37:33.712663 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:37:33.712987 392 external_mini_cluster.cc:1428] Setting key 36a5af9d7ded54cf857274c37fbee17a
Sep 27 02:37:33 dist-test-slave-grlk krb5kdc[19532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:37:33.721814 19616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:33.708779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55761 (local address 127.0.98.62:35123)
0927 02:37:33.708909 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:33.708914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:33.709760 (+ 846us) server_negotiation.cc:407] Connection header received
0927 02:37:33.710571 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:33.710574 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:33.710621 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:33.710691 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:33.711938 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:33.712633 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:33.714229 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:33.714662 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:33.714861 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:33.718114 (+ 3253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:33.718118 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:33.718121 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:37:33.718153 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:33.719556 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:33.720142 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:33.720145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:33.720146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:33.720225 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:33.720570 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:33.720572 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:33.720574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:33.720781 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:33.720867 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:33.721481 (+ 614us) server_negotiation.cc:299] Negotiation successful
0927 02:37:33.721635 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:37:33.722908 19770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:33.709174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35123 (local address 127.0.98.1:55761)
0927 02:37:33.709586 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:33.709624 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:33.710392 (+ 768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:33.710844 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:33.710850 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:33.711189 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:37:33.711758 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:33.711772 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:33.712790 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:33.712793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:33.714001 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:33.714011 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:33.714922 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:33.714925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:33.715094 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:33.715732 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:37:33.715750 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:37:33.717873 (+ 2123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:37:33.719693 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:37:33.719700 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:37:33.719719 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:37:33.720032 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:37:33.720306 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:37:33.720309 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:37:33.720311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:37:33.720472 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:37:33.720879 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:37:33.720887 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:37:33.721224 (+ 337us) client_negotiation.cc:769] Sending connection context
0927 02:37:33.721410 (+ 186us) client_negotiation.cc:240] Negotiation successful
0927 02:37:33.721841 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":112,"threads_started":1}
I20260927 02:37:33.724198 19768 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35123
I20260927 02:37:33.724478 19768 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 02:37:33.725095 19768 heartbeater.cc:511] Master 127.0.98.62:35123 requested a full tablet report, sending...
I20260927 02:37:33.727101 19577 ts_manager.cc:194] Registered new tserver with Master: eaf3a85f105b42e0b6bc5629ccacfc7e (127.0.98.1:34129)
I20260927 02:37:33.728377 19577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55761
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:37:33.742151 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35123
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:37:33.863744 19773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:37:33.864053 19773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:37:33.864097 19773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:37:33.868496 19773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:37:33.868578 19773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:37:33.868641 19773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:37:33.874198 19773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:37:33.875373 19773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:37:33.876459 19773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:37:33.886957 19781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:33.886950 19779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:33.887014 19773 server_base.cc:1034] running on GCE node
W20260927 02:37:33.886950 19778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:33.887732 19773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:37:33.888407 19773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:37:33.889571 19773 hybrid_clock.cc:648] HybridClock initialized: now 1790476653889545 us; error 40 us; skew 500 ppm
Sep 27 02:37:33 dist-test-slave-grlk krb5kdc[19532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:37:33.892299 19773 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:37:33.893296 19773 webserver.cc:460] Webserver started at http://127.0.98.2:44085/ using document root <none> and password file <none>
I20260927 02:37:33.893872 19773 fs_manager.cc:359] Metadata directory not provided
I20260927 02:37:33.893935 19773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:37:33.894127 19773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:37:33.896271 19773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-1/data/instance:
uuid: "bcc7255684884a67b0a4707d7bdfb239"
format_stamp: "Formatted at 2026-09-27 02:37:33 on dist-test-slave-grlk"
server_key: "99ad17cbf8ae771688665ea47159d7d4"
server_key_iv: "9367b2e6bb10f601830e9c9062bfb24d"
server_key_version: "encryptionkey@0"
I20260927 02:37:33.896821 19773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bcc7255684884a67b0a4707d7bdfb239"
format_stamp: "Formatted at 2026-09-27 02:37:33 on dist-test-slave-grlk"
server_key: "99ad17cbf8ae771688665ea47159d7d4"
server_key_iv: "9367b2e6bb10f601830e9c9062bfb24d"
server_key_version: "encryptionkey@0"
I20260927 02:37:33.901165 19773 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 02:37:33.904783 19788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:33.906569 19773 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:37:33.906741 19773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-1/wal
uuid: "bcc7255684884a67b0a4707d7bdfb239"
format_stamp: "Formatted at 2026-09-27 02:37:33 on dist-test-slave-grlk"
server_key: "99ad17cbf8ae771688665ea47159d7d4"
server_key_iv: "9367b2e6bb10f601830e9c9062bfb24d"
server_key_version: "encryptionkey@0"
I20260927 02:37:33.906900 19773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:37:33.922772 19773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:37:33.926810 19773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:37:33.927013 19773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:37:33.927670 19773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:37:33.928582 19773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:37:33.928630 19773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:33.928671 19773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:37:33.928685 19773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:33.939079 19773 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34535
I20260927 02:37:33.939103 19901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34535 every 8 connection(s)
I20260927 02:37:33.940356 19773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:37:33.950435 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 19773
I20260927 02:37:33.950604 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:37:33.950883 392 external_mini_cluster.cc:1428] Setting key b3873de1d2845d3ca24c748e5b73fdfe
Sep 27 02:37:33 dist-test-slave-grlk krb5kdc[19532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:37:33.956647 19616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:33.942770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58797 (local address 127.0.98.62:35123)
0927 02:37:33.942980 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:33.942984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:33.944198 (+ 1214us) server_negotiation.cc:407] Connection header received
0927 02:37:33.945858 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:33.945862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:33.945934 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:33.946024 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:33.947500 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:33.948096 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:33.949396 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:33.949630 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:33.949722 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:33.952866 (+ 3144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:33.952871 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:33.952875 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:37:33.952908 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:33.954470 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:33.955097 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:33.955100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:33.955101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:33.955172 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:33.955522 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:33.955525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:33.955527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:33.955755 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:33.955855 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:33.956347 (+ 492us) server_negotiation.cc:299] Negotiation successful
0927 02:37:33.956476 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 02:37:33.957552 19904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:33.943236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35123 (local address 127.0.98.2:58797)
0927 02:37:33.943992 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:33.944050 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:33.945606 (+ 1556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:33.946244 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:33.946252 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:33.946719 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:37:33.947342 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:33.947358 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:33.948215 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:33.948219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:33.949277 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:33.949288 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:33.949790 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:33.949794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:33.949974 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:33.950815 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:37:33.950839 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:37:33.952627 (+ 1788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:37:33.954607 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:37:33.954614 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:37:33.954632 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:37:33.954987 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:37:33.955250 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:37:33.955253 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:37:33.955255 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:37:33.955435 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:37:33.955840 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:37:33.955845 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:37:33.956116 (+ 271us) client_negotiation.cc:769] Sending connection context
0927 02:37:33.956293 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 02:37:33.956617 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":290,"threads_started":1}
I20260927 02:37:33.958798 19902 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35123
I20260927 02:37:33.959022 19902 heartbeater.cc:464] Registering TS with master...
I20260927 02:37:33.959514 19902 heartbeater.cc:511] Master 127.0.98.62:35123 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:37:33.960494 19577 ts_manager.cc:194] Registered new tserver with Master: bcc7255684884a67b0a4707d7bdfb239 (127.0.98.2:34535)
I20260927 02:37:33.961088 19577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58797
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:37:33.976029 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35123
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:37:34.079082 19907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:37:34.079294 19907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:37:34.079346 19907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:37:34.083247 19907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:37:34.083292 19907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:37:34.083365 19907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:37:34.088333 19907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:37:34.089195 19907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:37:34.090046 19907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:37:34.098753 19915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:34.098776 19912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:34.098873 19907 server_base.cc:1034] running on GCE node
W20260927 02:37:34.098770 19913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:34.099340 19907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:37:34.099896 19907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:37:34.101043 19907 hybrid_clock.cc:648] HybridClock initialized: now 1790476654101020 us; error 34 us; skew 500 ppm
Sep 27 02:37:34 dist-test-slave-grlk krb5kdc[19532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476654, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:37:34.103379 19907 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:37:34.104144 19907 webserver.cc:460] Webserver started at http://127.0.98.3:40993/ using document root <none> and password file <none>
I20260927 02:37:34.104625 19907 fs_manager.cc:359] Metadata directory not provided
I20260927 02:37:34.104686 19907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:37:34.104861 19907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:37:34.106693 19907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-2/data/instance:
uuid: "c5374b554bfd4b1fa6f814915f32e991"
format_stamp: "Formatted at 2026-09-27 02:37:34 on dist-test-slave-grlk"
server_key: "c89f42a81520ec8256ac7f5a037df867"
server_key_iv: "a7e437b86538db9de663d0adc1a9bf88"
server_key_version: "encryptionkey@0"
I20260927 02:37:34.107183 19907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c5374b554bfd4b1fa6f814915f32e991"
format_stamp: "Formatted at 2026-09-27 02:37:34 on dist-test-slave-grlk"
server_key: "c89f42a81520ec8256ac7f5a037df867"
server_key_iv: "a7e437b86538db9de663d0adc1a9bf88"
server_key_version: "encryptionkey@0"
I20260927 02:37:34.110600 19907 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:37:34.112854 19922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:34.113972 19907 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:37:34.114070 19907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-2/wal
uuid: "c5374b554bfd4b1fa6f814915f32e991"
format_stamp: "Formatted at 2026-09-27 02:37:34 on dist-test-slave-grlk"
server_key: "c89f42a81520ec8256ac7f5a037df867"
server_key_iv: "a7e437b86538db9de663d0adc1a9bf88"
server_key_version: "encryptionkey@0"
I20260927 02:37:34.114151 19907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:37:34.145618 19907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:37:34.148132 19907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:37:34.148300 19907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:37:34.148846 19907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:37:34.149621 19907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:37:34.149665 19907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:34.149717 19907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:37:34.149741 19907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:34.157630 19907 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40913
I20260927 02:37:34.157660 20035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40913 every 8 connection(s)
I20260927 02:37:34.158420 19907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:37:34.161131 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 19907
I20260927 02:37:34.161228 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:37:34.161473 392 external_mini_cluster.cc:1428] Setting key e2b568823f0ac6a87c8655702957d24d
Sep 27 02:37:34 dist-test-slave-grlk krb5kdc[19532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476654, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:37:34.170370 19616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:34.159973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53709 (local address 127.0.98.62:35123)
0927 02:37:34.160091 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:34.160094 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:34.160810 (+ 716us) server_negotiation.cc:407] Connection header received
0927 02:37:34.162123 (+ 1313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:34.162126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:34.162180 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:34.162250 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:34.163436 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:34.164032 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:34.165143 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:34.165365 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:34.165443 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:34.167529 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:34.167532 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:34.167534 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:37:34.167556 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:34.168850 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:34.169264 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:34.169266 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:34.169268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:34.169337 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:34.169574 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:34.169576 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:34.169577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:34.169727 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:34.169792 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:34.170162 (+ 370us) server_negotiation.cc:299] Negotiation successful
0927 02:37:34.170256 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 02:37:34.171047 20038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:34.160288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35123 (local address 127.0.98.3:53709)
0927 02:37:34.160685 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:34.160722 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:34.161922 (+ 1200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:34.162444 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:34.162453 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:34.162808 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:37:34.163291 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:34.163304 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:34.164135 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:34.164138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:34.165050 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:34.165059 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:34.165474 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:34.165477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:34.165605 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:34.166090 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:37:34.166105 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:37:34.167384 (+ 1279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:37:34.168954 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:37:34.168960 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:37:34.168974 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:37:34.169191 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:37:34.169400 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:37:34.169402 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:37:34.169403 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:37:34.169510 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:37:34.169792 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:37:34.169797 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:37:34.169990 (+ 193us) client_negotiation.cc:769] Sending connection context
0927 02:37:34.170121 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 02:37:34.170352 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":105,"threads_started":1}
I20260927 02:37:34.172016 20036 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35123
I20260927 02:37:34.172204 20036 heartbeater.cc:464] Registering TS with master...
I20260927 02:37:34.172603 20036 heartbeater.cc:511] Master 127.0.98.62:35123 requested a full tablet report, sending...
I20260927 02:37:34.173484 19577 ts_manager.cc:194] Registered new tserver with Master: c5374b554bfd4b1fa6f814915f32e991 (127.0.98.3:40913)
I20260927 02:37:34.174067 19577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53709
I20260927 02:37:34.174952 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:37:34.182904 19616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:34.176612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60370 (local address 127.0.98.62:35123)
0927 02:37:34.176716 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:34.176720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:34.176882 (+ 162us) server_negotiation.cc:407] Connection header received
0927 02:37:34.177002 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:34.177004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:34.177044 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:34.177104 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:34.177795 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:34.178263 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:34.179032 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:34.179238 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:34.179317 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:34.180400 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:34.180403 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:34.180405 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:37:34.180424 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:34.181462 (+ 1038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:34.181820 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:34.181824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:34.181825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:34.181891 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:34.182137 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:34.182139 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:34.182140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:34.182363 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:34.182478 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:34.182673 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 02:37:34.182769 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 02:37:34.185120 19577 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60370:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:37:34.187934 19577 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:37:34.200416 20047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:34.195079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34129 (local address 127.0.0.1:58838)
0927 02:37:34.195526 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:34.195573 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:34.195692 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:34.196401 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:34.196407 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:34.196427 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:37:34.196540 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:34.196547 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:34.197961 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:34.197965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:34.199001 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:34.199012 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:34.199927 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:34.199930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:34.200032 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:34.200067 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 02:37:34.200164 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 02:37:34.200275 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":22,"thread_start_us":70,"threads_started":1}
I20260927 02:37:34.200439 20049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:34.195220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58838 (local address 127.0.98.1:34129)
0927 02:37:34.195540 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:34.195555 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:34.195652 (+ 97us) server_negotiation.cc:407] Connection header received
0927 02:37:34.195872 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:34.195878 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:34.196079 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:34.196224 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:37:34.196660 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:34.197835 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:34.199138 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:34.199817 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:34.199914 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:34.200048 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:34.200210 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 02:37:34.200300 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":56,"threads_started":1}
I20260927 02:37:34.201066 20050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:34.195184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40913 (local address 127.0.0.1:55566)
0927 02:37:34.196023 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:34.196035 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:34.196121 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:34.196693 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:34.196696 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:34.196713 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:37:34.196789 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:34.196794 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:34.198095 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:34.198099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:34.199347 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:34.199357 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:34.200535 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:34.200539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:34.200685 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:34.200719 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 02:37:34.200785 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 02:37:34.200906 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":784,"mutex_wait_us":147,"thread_start_us":76,"threads_started":1}
I20260927 02:37:34.201109 20051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:34.195787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55566 (local address 127.0.98.3:40913)
0927 02:37:34.196168 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:34.196179 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:34.196198 (+ 19us) server_negotiation.cc:407] Connection header received
0927 02:37:34.196257 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:34.196262 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:34.196428 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:34.196560 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:37:34.196917 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:34.197988 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:34.199484 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:34.200258 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:34.200354 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:34.200473 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:34.200871 (+ 398us) server_negotiation.cc:299] Negotiation successful
0927 02:37:34.200977 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":132,"threads_started":1}
I20260927 02:37:34.201210 20052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:34.195965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50866 (local address 127.0.98.2:34535)
0927 02:37:34.196459 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:34.196471 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:34.196777 (+ 306us) server_negotiation.cc:407] Connection header received
0927 02:37:34.196854 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:34.196859 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:34.197040 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:34.197158 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:37:34.197358 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:34.198157 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:34.199496 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:34.200258 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:34.200363 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:34.200473 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:34.200997 (+ 524us) server_negotiation.cc:299] Negotiation successful
0927 02:37:34.201080 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":119,"threads_started":1}
I20260927 02:37:34.201408 20048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:34.195752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34535 (local address 127.0.0.1:50866)
0927 02:37:34.196580 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:34.196591 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:34.196669 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:34.197155 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:34.197158 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:34.197173 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:37:34.197262 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:34.197268 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:34.198288 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:34.198290 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:34.199386 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:34.199393 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:34.200535 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:34.200539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:34.200674 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:34.200724 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 02:37:34.200784 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 02:37:34.200871 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":776,"mutex_wait_us":33,"thread_start_us":86,"threads_started":1}
I20260927 02:37:34.203544 19702 tablet_service.cc:1467] Processing CreateTablet for tablet 698fdc7a636844fc8656ad3cc11d0d4e (DEFAULT_TABLE table=test-table [id=b63329318b444274abcd321f8e62ccd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:37:34.203814 19970 tablet_service.cc:1467] Processing CreateTablet for tablet 698fdc7a636844fc8656ad3cc11d0d4e (DEFAULT_TABLE table=test-table [id=b63329318b444274abcd321f8e62ccd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:37:34.204627 19970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 698fdc7a636844fc8656ad3cc11d0d4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:37:34.204627 19702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 698fdc7a636844fc8656ad3cc11d0d4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:37:34.205674 19836 tablet_service.cc:1467] Processing CreateTablet for tablet 698fdc7a636844fc8656ad3cc11d0d4e (DEFAULT_TABLE table=test-table [id=b63329318b444274abcd321f8e62ccd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:37:34.206431 19836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 698fdc7a636844fc8656ad3cc11d0d4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:37:34.211495 20054 tablet_bootstrap.cc:492] T 698fdc7a636844fc8656ad3cc11d0d4e P eaf3a85f105b42e0b6bc5629ccacfc7e: Bootstrap starting.
I20260927 02:37:34.212110 20053 tablet_bootstrap.cc:492] T 698fdc7a636844fc8656ad3cc11d0d4e P c5374b554bfd4b1fa6f814915f32e991: Bootstrap starting.
I20260927 02:37:34.212317 20055 tablet_bootstrap.cc:492] T 698fdc7a636844fc8656ad3cc11d0d4e P bcc7255684884a67b0a4707d7bdfb239: Bootstrap starting.
I20260927 02:37:34.214178 20054 tablet_bootstrap.cc:654] T 698fdc7a636844fc8656ad3cc11d0d4e P eaf3a85f105b42e0b6bc5629ccacfc7e: Neither blocks nor log segments found. Creating new log.
I20260927 02:37:34.214294 20053 tablet_bootstrap.cc:654] T 698fdc7a636844fc8656ad3cc11d0d4e P c5374b554bfd4b1fa6f814915f32e991: Neither blocks nor log segments found. Creating new log.
I20260927 02:37:34.214294 20055 tablet_bootstrap.cc:654] T 698fdc7a636844fc8656ad3cc11d0d4e P bcc7255684884a67b0a4707d7bdfb239: Neither blocks nor log segments found. Creating new log.
I20260927 02:37:34.215147 20054 log.cc:824] T 698fdc7a636844fc8656ad3cc11d0d4e P eaf3a85f105b42e0b6bc5629ccacfc7e: Log is configured to *not* fsync() on all Append() calls
I20260927 02:37:34.215474 20055 log.cc:824] T 698fdc7a636844fc8656ad3cc11d0d4e P bcc7255684884a67b0a4707d7bdfb239: Log is configured to *not* fsync() on all Append() calls
I20260927 02:37:34.215474 20053 log.cc:824] T 698fdc7a636844fc8656ad3cc11d0d4e P c5374b554bfd4b1fa6f814915f32e991: Log is configured to *not* fsync() on all Append() calls
I20260927 02:37:34.216856 20054 tablet_bootstrap.cc:492] T 698fdc7a636844fc8656ad3cc11d0d4e P eaf3a85f105b42e0b6bc5629ccacfc7e: No bootstrap required, opened a new log
I20260927 02:37:34.217067 20054 ts_tablet_manager.cc:1397] T 698fdc7a636844fc8656ad3cc11d0d4e P eaf3a85f105b42e0b6bc5629ccacfc7e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 02:37:34.217110 20053 tablet_bootstrap.cc:492] T 698fdc7a636844fc8656ad3cc11d0d4e P c5374b554bfd4b1fa6f814915f32e991: No bootstrap required, opened a new log
I20260927 02:37:34.217267 20053 ts_tablet_manager.cc:1397] T 698fdc7a636844fc8656ad3cc11d0d4e P c5374b554bfd4b1fa6f814915f32e991: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 02:37:34.217258 20055 tablet_bootstrap.cc:492] T 698fdc7a636844fc8656ad3cc11d0d4e P bcc7255684884a67b0a4707d7bdfb239: No bootstrap required, opened a new log
I20260927 02:37:34.217430 20055 ts_tablet_manager.cc:1397] T 698fdc7a636844fc8656ad3cc11d0d4e P bcc7255684884a67b0a4707d7bdfb239: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 02:37:34.220059 20053 raft_consensus.cc:348] T 698fdc7a636844fc8656ad3cc11d0d4e P c5374b554bfd4b1fa6f814915f32e991 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf3a85f105b42e0b6bc5629ccacfc7e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34129 } } peers { permanent_uuid: "c5374b554bfd4b1fa6f814915f32e991" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40913 } } peers { permanent_uuid: "bcc7255684884a67b0a4707d7bdfb239" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34535 } }
I20260927 02:37:34.220256 20053 raft_consensus.cc:374] T 698fdc7a636844fc8656ad3cc11d0d4e P c5374b554bfd4b1fa6f814915f32e991 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:37:34.220304 20053 raft_consensus.cc:729] T 698fdc7a636844fc8656ad3cc11d0d4e P c5374b554bfd4b1fa6f814915f32e991 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5374b554bfd4b1fa6f814915f32e991, State: Initialized, Role: FOLLOWER
I20260927 02:37:34.220664 20053 consensus_queue.cc:260] T 698fdc7a636844fc8656ad3cc11d0d4e P c5374b554bfd4b1fa6f814915f32e991 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf3a85f105b42e0b6bc5629ccacfc7e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34129 } } peers { permanent_uuid: "c5374b554bfd4b1fa6f814915f32e991" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40913 } } peers { permanent_uuid: "bcc7255684884a67b0a4707d7bdfb239" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34535 } }
I20260927 02:37:34.220641 20054 raft_consensus.cc:348] T 698fdc7a636844fc8656ad3cc11d0d4e P eaf3a85f105b42e0b6bc5629ccacfc7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf3a85f105b42e0b6bc5629ccacfc7e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34129 } } peers { permanent_uuid: "c5374b554bfd4b1fa6f814915f32e991" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40913 } } peers { permanent_uuid: "bcc7255684884a67b0a4707d7bdfb239" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34535 } }
I20260927 02:37:34.220889 20054 raft_consensus.cc:374] T 698fdc7a636844fc8656ad3cc11d0d4e P eaf3a85f105b42e0b6bc5629ccacfc7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:37:34.220958 20054 raft_consensus.cc:729] T 698fdc7a636844fc8656ad3cc11d0d4e P eaf3a85f105b42e0b6bc5629ccacfc7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf3a85f105b42e0b6bc5629ccacfc7e, State: Initialized, Role: FOLLOWER
I20260927 02:37:34.220793 20055 raft_consensus.cc:348] T 698fdc7a636844fc8656ad3cc11d0d4e P bcc7255684884a67b0a4707d7bdfb239 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf3a85f105b42e0b6bc5629ccacfc7e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34129 } } peers { permanent_uuid: "c5374b554bfd4b1fa6f814915f32e991" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40913 } } peers { permanent_uuid: "bcc7255684884a67b0a4707d7bdfb239" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34535 } }
I20260927 02:37:34.221026 20055 raft_consensus.cc:374] T 698fdc7a636844fc8656ad3cc11d0d4e P bcc7255684884a67b0a4707d7bdfb239 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:37:34.221076 20055 raft_consensus.cc:729] T 698fdc7a636844fc8656ad3cc11d0d4e P bcc7255684884a67b0a4707d7bdfb239 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcc7255684884a67b0a4707d7bdfb239, State: Initialized, Role: FOLLOWER
I20260927 02:37:34.221406 20055 consensus_queue.cc:260] T 698fdc7a636844fc8656ad3cc11d0d4e P bcc7255684884a67b0a4707d7bdfb239 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf3a85f105b42e0b6bc5629ccacfc7e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34129 } } peers { permanent_uuid: "c5374b554bfd4b1fa6f814915f32e991" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40913 } } peers { permanent_uuid: "bcc7255684884a67b0a4707d7bdfb239" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34535 } }
I20260927 02:37:34.221396 20054 consensus_queue.cc:260] T 698fdc7a636844fc8656ad3cc11d0d4e P eaf3a85f105b42e0b6bc5629ccacfc7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf3a85f105b42e0b6bc5629ccacfc7e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34129 } } peers { permanent_uuid: "c5374b554bfd4b1fa6f814915f32e991" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40913 } } peers { permanent_uuid: "bcc7255684884a67b0a4707d7bdfb239" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34535 } }
I20260927 02:37:34.221504 20053 ts_tablet_manager.cc:1428] T 698fdc7a636844fc8656ad3cc11d0d4e P c5374b554bfd4b1fa6f814915f32e991: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260927 02:37:34.221798 20036 heartbeater.cc:503] Master 127.0.98.62:35123 was elected leader, sending a full tablet report...
I20260927 02:37:34.222208 19902 heartbeater.cc:503] Master 127.0.98.62:35123 was elected leader, sending a full tablet report...
I20260927 02:37:34.222263 20055 ts_tablet_manager.cc:1428] T 698fdc7a636844fc8656ad3cc11d0d4e P bcc7255684884a67b0a4707d7bdfb239: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260927 02:37:34.222254 20054 ts_tablet_manager.cc:1428] T 698fdc7a636844fc8656ad3cc11d0d4e P eaf3a85f105b42e0b6bc5629ccacfc7e: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260927 02:37:34.222539 19768 heartbeater.cc:503] Master 127.0.98.62:35123 was elected leader, sending a full tablet report...
I20260927 02:37:34.234794 20059 raft_consensus.cc:482] T 698fdc7a636844fc8656ad3cc11d0d4e P c5374b554bfd4b1fa6f814915f32e991 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:37:34.234920 20059 raft_consensus.cc:504] T 698fdc7a636844fc8656ad3cc11d0d4e P c5374b554bfd4b1fa6f814915f32e991 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf3a85f105b42e0b6bc5629ccacfc7e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34129 } } peers { permanent_uuid: "c5374b554bfd4b1fa6f814915f32e991" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40913 } } peers { permanent_uuid: "bcc7255684884a67b0a4707d7bdfb239" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34535 } }
I20260927 02:37:34.235735 20059 leader_election.cc:288] T 698fdc7a636844fc8656ad3cc11d0d4e P c5374b554bfd4b1fa6f814915f32e991 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf3a85f105b42e0b6bc5629ccacfc7e (127.0.98.1:34129), bcc7255684884a67b0a4707d7bdfb239 (127.0.98.2:34535)
I20260927 02:37:34.239219 20049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:34.235959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39023 (local address 127.0.98.1:34129)
0927 02:37:34.236076 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:34.236080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:34.236221 (+ 141us) server_negotiation.cc:407] Connection header received
0927 02:37:34.236330 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:34.236333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:34.236368 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:34.236425 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:37:34.236638 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:34.237164 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:34.238245 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:34.238747 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:34.238837 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:34.238893 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:34.239054 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 02:37:34.239113 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 02:37:34.239261 20038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:34.235880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34129 (local address 127.0.98.3:39023)
0927 02:37:34.236157 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:34.236168 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:34.236246 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:34.236439 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:34.236442 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:34.236455 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:37:34.236554 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:34.236558 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:34.237262 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:34.237265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:34.238133 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:34.238143 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:34.238850 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:34.238853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:34.238952 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:34.238972 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 02:37:34.239033 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 02:37:34.239100 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":38}
I20260927 02:37:34.239256 20052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:34.236008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44477 (local address 127.0.98.2:34535)
0927 02:37:34.236111 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:34.236115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:34.236265 (+ 150us) server_negotiation.cc:407] Connection header received
0927 02:37:34.236393 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:34.236395 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:34.236431 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:34.236485 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:37:34.236714 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:34.237242 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:34.238297 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:34.238792 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:34.238884 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:34.238937 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:34.239092 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 02:37:34.239150 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 02:37:34.239288 20062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:34.235942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34535 (local address 127.0.98.3:44477)
0927 02:37:34.236176 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:34.236188 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:34.236310 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:34.236502 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:34.236505 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:34.236519 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:37:34.236619 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:34.236623 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:34.237337 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:34.237340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:34.238202 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:34.238213 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:34.238895 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:34.238899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:34.238994 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:34.239010 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 02:37:34.239071 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 02:37:34.239133 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":6,"thread_start_us":81,"threads_started":1}
I20260927 02:37:34.239763 19856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "698fdc7a636844fc8656ad3cc11d0d4e" candidate_uuid: "c5374b554bfd4b1fa6f814915f32e991" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcc7255684884a67b0a4707d7bdfb239" is_pre_election: true
I20260927 02:37:34.239796 19722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "698fdc7a636844fc8656ad3cc11d0d4e" candidate_uuid: "c5374b554bfd4b1fa6f814915f32e991" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf3a85f105b42e0b6bc5629ccacfc7e" is_pre_election: true
I20260927 02:37:34.239971 19856 raft_consensus.cc:2454] T 698fdc7a636844fc8656ad3cc11d0d4e P bcc7255684884a67b0a4707d7bdfb239 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5374b554bfd4b1fa6f814915f32e991 in term 0.
I20260927 02:37:34.239995 19722 raft_consensus.cc:2454] T 698fdc7a636844fc8656ad3cc11d0d4e P eaf3a85f105b42e0b6bc5629ccacfc7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5374b554bfd4b1fa6f814915f32e991 in term 0.
I20260927 02:37:34.240360 19924 leader_election.cc:302] T 698fdc7a636844fc8656ad3cc11d0d4e P c5374b554bfd4b1fa6f814915f32e991 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: bcc7255684884a67b0a4707d7bdfb239, c5374b554bfd4b1fa6f814915f32e991, eaf3a85f105b42e0b6bc5629ccacfc7e; no voters:
I20260927 02:37:34.240546 20059 raft_consensus.cc:2781] T 698fdc7a636844fc8656ad3cc11d0d4e P c5374b554bfd4b1fa6f814915f32e991 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:37:34.240612 20059 raft_consensus.cc:482] T 698fdc7a636844fc8656ad3cc11d0d4e P c5374b554bfd4b1fa6f814915f32e991 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:37:34.240656 20059 raft_consensus.cc:3037] T 698fdc7a636844fc8656ad3cc11d0d4e P c5374b554bfd4b1fa6f814915f32e991 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:37:34.241552 20059 raft_consensus.cc:504] T 698fdc7a636844fc8656ad3cc11d0d4e P c5374b554bfd4b1fa6f814915f32e991 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf3a85f105b42e0b6bc5629ccacfc7e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34129 } } peers { permanent_uuid: "c5374b554bfd4b1fa6f814915f32e991" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40913 } } peers { permanent_uuid: "bcc7255684884a67b0a4707d7bdfb239" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34535 } }
I20260927 02:37:34.241840 20059 leader_election.cc:288] T 698fdc7a636844fc8656ad3cc11d0d4e P c5374b554bfd4b1fa6f814915f32e991 [CANDIDATE]: Term 1 election: Requested vote from peers eaf3a85f105b42e0b6bc5629ccacfc7e (127.0.98.1:34129), bcc7255684884a67b0a4707d7bdfb239 (127.0.98.2:34535)
I20260927 02:37:34.242146 19722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "698fdc7a636844fc8656ad3cc11d0d4e" candidate_uuid: "c5374b554bfd4b1fa6f814915f32e991" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf3a85f105b42e0b6bc5629ccacfc7e"
I20260927 02:37:34.242239 19722 raft_consensus.cc:3037] T 698fdc7a636844fc8656ad3cc11d0d4e P eaf3a85f105b42e0b6bc5629ccacfc7e [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:37:34.243080 19722 raft_consensus.cc:2454] T 698fdc7a636844fc8656ad3cc11d0d4e P eaf3a85f105b42e0b6bc5629ccacfc7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5374b554bfd4b1fa6f814915f32e991 in term 1.
I20260927 02:37:34.243299 19856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "698fdc7a636844fc8656ad3cc11d0d4e" candidate_uuid: "c5374b554bfd4b1fa6f814915f32e991" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcc7255684884a67b0a4707d7bdfb239"
I20260927 02:37:34.243364 19925 leader_election.cc:302] T 698fdc7a636844fc8656ad3cc11d0d4e P c5374b554bfd4b1fa6f814915f32e991 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5374b554bfd4b1fa6f814915f32e991, eaf3a85f105b42e0b6bc5629ccacfc7e; no voters:
I20260927 02:37:34.243408 19856 raft_consensus.cc:3037] T 698fdc7a636844fc8656ad3cc11d0d4e P bcc7255684884a67b0a4707d7bdfb239 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:37:34.243518 20059 raft_consensus.cc:2781] T 698fdc7a636844fc8656ad3cc11d0d4e P c5374b554bfd4b1fa6f814915f32e991 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:37:34.243702 20059 raft_consensus.cc:686] T 698fdc7a636844fc8656ad3cc11d0d4e P c5374b554bfd4b1fa6f814915f32e991 [term 1 LEADER]: Becoming Leader. State: Replica: c5374b554bfd4b1fa6f814915f32e991, State: Running, Role: LEADER
I20260927 02:37:34.243974 20059 consensus_queue.cc:237] T 698fdc7a636844fc8656ad3cc11d0d4e P c5374b554bfd4b1fa6f814915f32e991 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf3a85f105b42e0b6bc5629ccacfc7e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34129 } } peers { permanent_uuid: "c5374b554bfd4b1fa6f814915f32e991" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40913 } } peers { permanent_uuid: "bcc7255684884a67b0a4707d7bdfb239" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34535 } }
I20260927 02:37:34.244460 19856 raft_consensus.cc:2454] T 698fdc7a636844fc8656ad3cc11d0d4e P bcc7255684884a67b0a4707d7bdfb239 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5374b554bfd4b1fa6f814915f32e991 in term 1.
I20260927 02:37:34.246526 19576 catalog_manager.cc:5495] T 698fdc7a636844fc8656ad3cc11d0d4e P c5374b554bfd4b1fa6f814915f32e991 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5374b554bfd4b1fa6f814915f32e991 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c5374b554bfd4b1fa6f814915f32e991" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf3a85f105b42e0b6bc5629ccacfc7e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5374b554bfd4b1fa6f814915f32e991" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcc7255684884a67b0a4707d7bdfb239" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34535 } health_report { overall_health: UNKNOWN } } }
I20260927 02:37:34.255687 20051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:34.252557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55582 (local address 127.0.98.3:40913)
0927 02:37:34.252673 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:34.252676 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:34.252689 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:37:34.252804 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:34.252807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:34.252862 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:34.252962 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:37:34.253185 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:34.253643 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:34.254460 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:34.254665 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:34.254744 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:34.255005 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:37:34.255346 (+ 341us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:37:34.255442 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:34.255509 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 02:37:34.255556 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 02:37:34.260528 19856 raft_consensus.cc:1261] T 698fdc7a636844fc8656ad3cc11d0d4e P bcc7255684884a67b0a4707d7bdfb239 [term 1 FOLLOWER]: Refusing update from remote peer c5374b554bfd4b1fa6f814915f32e991: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:37:34.260527 19722 raft_consensus.cc:1261] T 698fdc7a636844fc8656ad3cc11d0d4e P eaf3a85f105b42e0b6bc5629ccacfc7e [term 1 FOLLOWER]: Refusing update from remote peer c5374b554bfd4b1fa6f814915f32e991: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:37:34.261134 20059 consensus_queue.cc:1035] T 698fdc7a636844fc8656ad3cc11d0d4e P c5374b554bfd4b1fa6f814915f32e991 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf3a85f105b42e0b6bc5629ccacfc7e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:37:34.261302 20063 consensus_queue.cc:1035] T 698fdc7a636844fc8656ad3cc11d0d4e P c5374b554bfd4b1fa6f814915f32e991 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcc7255684884a67b0a4707d7bdfb239" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:37:34.274683 19576 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:37:34.274847 19576 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:37:34.277182 19576 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fd469a06e1f647d5bd336e888887529f: Sending DeleteTablet for 3 replicas of tablet 698fdc7a636844fc8656ad3cc11d0d4e
I20260927 02:37:34.277931 19702 tablet_service.cc:1514] Processing DeleteTablet for tablet 698fdc7a636844fc8656ad3cc11d0d4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:37:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58838
I20260927 02:37:34.277956 19970 tablet_service.cc:1514] Processing DeleteTablet for tablet 698fdc7a636844fc8656ad3cc11d0d4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:37:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55566
I20260927 02:37:34.278054 19836 tablet_service.cc:1514] Processing DeleteTablet for tablet 698fdc7a636844fc8656ad3cc11d0d4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:37:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50866
I20260927 02:37:34.278436 20077 tablet_replica.cc:330] T 698fdc7a636844fc8656ad3cc11d0d4e P bcc7255684884a67b0a4707d7bdfb239: stopping tablet replica
I20260927 02:37:34.278479 20076 tablet_replica.cc:330] T 698fdc7a636844fc8656ad3cc11d0d4e P eaf3a85f105b42e0b6bc5629ccacfc7e: stopping tablet replica
I20260927 02:37:34.278740 20077 raft_consensus.cc:2229] T 698fdc7a636844fc8656ad3cc11d0d4e P bcc7255684884a67b0a4707d7bdfb239 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:37:34.278743 20076 raft_consensus.cc:2229] T 698fdc7a636844fc8656ad3cc11d0d4e P eaf3a85f105b42e0b6bc5629ccacfc7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:37:34.278769 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 19639
I20260927 02:37:34.279009 20076 raft_consensus.cc:2258] T 698fdc7a636844fc8656ad3cc11d0d4e P eaf3a85f105b42e0b6bc5629ccacfc7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:37:34.279125 20077 raft_consensus.cc:2258] T 698fdc7a636844fc8656ad3cc11d0d4e P bcc7255684884a67b0a4707d7bdfb239 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:37:34.279775 20075 tablet_replica.cc:330] T 698fdc7a636844fc8656ad3cc11d0d4e P c5374b554bfd4b1fa6f814915f32e991: stopping tablet replica
I20260927 02:37:34.280020 20075 raft_consensus.cc:2229] T 698fdc7a636844fc8656ad3cc11d0d4e P c5374b554bfd4b1fa6f814915f32e991 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:37:34.280341 20075 raft_consensus.cc:2258] T 698fdc7a636844fc8656ad3cc11d0d4e P c5374b554bfd4b1fa6f814915f32e991 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:37:34.281033 20077 ts_tablet_manager.cc:1905] T 698fdc7a636844fc8656ad3cc11d0d4e P bcc7255684884a67b0a4707d7bdfb239: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:37:34.281738 20075 ts_tablet_manager.cc:1905] T 698fdc7a636844fc8656ad3cc11d0d4e P c5374b554bfd4b1fa6f814915f32e991: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:37:34.283741 20077 ts_tablet_manager.cc:1918] T 698fdc7a636844fc8656ad3cc11d0d4e P bcc7255684884a67b0a4707d7bdfb239: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:37:34.283883 20077 log.cc:1196] T 698fdc7a636844fc8656ad3cc11d0d4e P bcc7255684884a67b0a4707d7bdfb239: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-1/wal/wals/698fdc7a636844fc8656ad3cc11d0d4e
I20260927 02:37:34.284243 20077 ts_tablet_manager.cc:1939] T 698fdc7a636844fc8656ad3cc11d0d4e P bcc7255684884a67b0a4707d7bdfb239: Deleting consensus metadata
I20260927 02:37:34.284492 20075 ts_tablet_manager.cc:1918] T 698fdc7a636844fc8656ad3cc11d0d4e P c5374b554bfd4b1fa6f814915f32e991: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:37:34.284578 20075 log.cc:1196] T 698fdc7a636844fc8656ad3cc11d0d4e P c5374b554bfd4b1fa6f814915f32e991: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476410933508-392-0/minicluster-data/ts-2/wal/wals/698fdc7a636844fc8656ad3cc11d0d4e
I20260927 02:37:34.284865 20075 ts_tablet_manager.cc:1939] T 698fdc7a636844fc8656ad3cc11d0d4e P c5374b554bfd4b1fa6f814915f32e991: Deleting consensus metadata
I20260927 02:37:34.285576 19562 catalog_manager.cc:4841] TS bcc7255684884a67b0a4707d7bdfb239 (127.0.98.2:34535): tablet 698fdc7a636844fc8656ad3cc11d0d4e (table test-table [id=b63329318b444274abcd321f8e62ccd0]) successfully deleted
I20260927 02:37:34.285838 19561 catalog_manager.cc:4841] TS c5374b554bfd4b1fa6f814915f32e991 (127.0.98.3:40913): tablet 698fdc7a636844fc8656ad3cc11d0d4e (table test-table [id=b63329318b444274abcd321f8e62ccd0]) successfully deleted
W20260927 02:37:34.286854 19563 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34129 (error 108)
I20260927 02:37:34.287565 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 19773
I20260927 02:37:34.287789 20050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:34.287511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34129 (local address 127.0.0.1:58842)
0927 02:37:34.287647 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:34.287713 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 02:37:34.289433 19563 catalog_manager.cc:4584] TS eaf3a85f105b42e0b6bc5629ccacfc7e (127.0.98.1:34129): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 698fdc7a636844fc8656ad3cc11d0d4e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34129: connect: Connection refused (error 111)
I20260927 02:37:34.292965 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 19907
I20260927 02:37:34.297627 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 19545
2026-09-27T02:37:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2071 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T02:37:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:37:34Z Disabled control of system clock
I20260927 02:37:34.325511 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41899
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41899
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 02:37:34.430441 20082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:37:34.430662 20082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:37:34.430693 20082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:37:34.434566 20082 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:37:34.434609 20082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:37:34.434623 20082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:37:34.434638 20082 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:37:34.434656 20082 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:37:34.439100 20082 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41899
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:37:34.439939 20082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:37:34.441157 20082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:37:34.448772 20087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:34.448810 20088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:34.448854 20082 server_base.cc:1034] running on GCE node
W20260927 02:37:34.448802 20090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:34.449582 20082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:37:34.450583 20082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:37:34.451725 20082 hybrid_clock.cc:648] HybridClock initialized: now 1790476654451704 us; error 39 us; skew 500 ppm
I20260927 02:37:34.453414 20082 webserver.cc:460] Webserver started at http://127.0.98.62:45203/ using document root <none> and password file <none>
I20260927 02:37:34.453933 20082 fs_manager.cc:359] Metadata directory not provided
I20260927 02:37:34.453996 20082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:37:34.454175 20082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:37:34.456122 20082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "244f664509414112bc0d7f7cdb90c49a"
format_stamp: "Formatted at 2026-09-27 02:37:34 on dist-test-slave-grlk"
server_key: "3d2907d2f9cfcbd6b3142863e5c03436"
server_key_iv: "c2fbcaa1cfcf7bea2866743372ef5b67"
server_key_version: "encryptionkey@0"
I20260927 02:37:34.456619 20082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "244f664509414112bc0d7f7cdb90c49a"
format_stamp: "Formatted at 2026-09-27 02:37:34 on dist-test-slave-grlk"
server_key: "3d2907d2f9cfcbd6b3142863e5c03436"
server_key_iv: "c2fbcaa1cfcf7bea2866743372ef5b67"
server_key_version: "encryptionkey@0"
I20260927 02:37:34.459983 20082 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.001s
I20260927 02:37:34.462198 20096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:34.463290 20082 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:37:34.463405 20082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "244f664509414112bc0d7f7cdb90c49a"
format_stamp: "Formatted at 2026-09-27 02:37:34 on dist-test-slave-grlk"
server_key: "3d2907d2f9cfcbd6b3142863e5c03436"
server_key_iv: "c2fbcaa1cfcf7bea2866743372ef5b67"
server_key_version: "encryptionkey@0"
I20260927 02:37:34.463488 20082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:37:34.478533 20082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:37:34.479104 20082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:37:34.479259 20082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:37:34.486032 20082 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41899
I20260927 02:37:34.486037 20148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41899 every 8 connection(s)
I20260927 02:37:34.486954 20082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:37:34.489982 20149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:37:34.490304 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 20082
I20260927 02:37:34.490432 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:37:34.490772 392 external_mini_cluster.cc:1428] Setting key 17032df8d3e5e1fc993e0249cfea1e1c
I20260927 02:37:34.496024 20149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 244f664509414112bc0d7f7cdb90c49a: Bootstrap starting.
I20260927 02:37:34.498492 20149 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 244f664509414112bc0d7f7cdb90c49a: Neither blocks nor log segments found. Creating new log.
I20260927 02:37:34.499265 20149 log.cc:824] T 00000000000000000000000000000000 P 244f664509414112bc0d7f7cdb90c49a: Log is configured to *not* fsync() on all Append() calls
I20260927 02:37:34.499934 20152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:34.492175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53696 (local address 127.0.98.62:41899)
0927 02:37:34.492885 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:34.492910 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:34.492986 (+ 76us) server_negotiation.cc:407] Connection header received
0927 02:37:34.493677 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:34.493695 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:34.494213 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:34.494528 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:34.494872 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:34.495814 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:34.496918 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:34.497400 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:34.497511 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:34.498277 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:34.498282 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:34.498293 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:37:34.498677 (+ 384us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:34.498733 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:37:34.498754 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:34.498828 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:34.498964 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 02:37:34.499206 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":142,"threads_started":1}
I20260927 02:37:34.501058 20149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 244f664509414112bc0d7f7cdb90c49a: No bootstrap required, opened a new log
I20260927 02:37:34.504236 20149 raft_consensus.cc:348] T 00000000000000000000000000000000 P 244f664509414112bc0d7f7cdb90c49a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "244f664509414112bc0d7f7cdb90c49a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41899 } }
I20260927 02:37:34.504448 20149 raft_consensus.cc:374] T 00000000000000000000000000000000 P 244f664509414112bc0d7f7cdb90c49a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:37:34.504503 20149 raft_consensus.cc:729] T 00000000000000000000000000000000 P 244f664509414112bc0d7f7cdb90c49a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244f664509414112bc0d7f7cdb90c49a, State: Initialized, Role: FOLLOWER
I20260927 02:37:34.504869 20149 consensus_queue.cc:260] T 00000000000000000000000000000000 P 244f664509414112bc0d7f7cdb90c49a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "244f664509414112bc0d7f7cdb90c49a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41899 } }
I20260927 02:37:34.505018 20149 raft_consensus.cc:388] T 00000000000000000000000000000000 P 244f664509414112bc0d7f7cdb90c49a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:37:34.505076 20149 raft_consensus.cc:482] T 00000000000000000000000000000000 P 244f664509414112bc0d7f7cdb90c49a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:37:34.505169 20149 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 244f664509414112bc0d7f7cdb90c49a [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:37:34.505942 20149 raft_consensus.cc:504] T 00000000000000000000000000000000 P 244f664509414112bc0d7f7cdb90c49a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "244f664509414112bc0d7f7cdb90c49a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41899 } }
I20260927 02:37:34.506232 20149 leader_election.cc:302] T 00000000000000000000000000000000 P 244f664509414112bc0d7f7cdb90c49a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 244f664509414112bc0d7f7cdb90c49a; no voters:
I20260927 02:37:34.506500 20149 leader_election.cc:288] T 00000000000000000000000000000000 P 244f664509414112bc0d7f7cdb90c49a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:37:34.506590 20154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 244f664509414112bc0d7f7cdb90c49a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:37:34.506790 20154 raft_consensus.cc:686] T 00000000000000000000000000000000 P 244f664509414112bc0d7f7cdb90c49a [term 1 LEADER]: Becoming Leader. State: Replica: 244f664509414112bc0d7f7cdb90c49a, State: Running, Role: LEADER
I20260927 02:37:34.507052 20154 consensus_queue.cc:237] T 00000000000000000000000000000000 P 244f664509414112bc0d7f7cdb90c49a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "244f664509414112bc0d7f7cdb90c49a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41899 } }
I20260927 02:37:34.507597 20149 sys_catalog.cc:564] T 00000000000000000000000000000000 P 244f664509414112bc0d7f7cdb90c49a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:37:34.508630 20155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 244f664509414112bc0d7f7cdb90c49a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "244f664509414112bc0d7f7cdb90c49a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "244f664509414112bc0d7f7cdb90c49a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41899 } } }
I20260927 02:37:34.508633 20156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 244f664509414112bc0d7f7cdb90c49a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 244f664509414112bc0d7f7cdb90c49a. Latest consensus state: current_term: 1 leader_uuid: "244f664509414112bc0d7f7cdb90c49a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "244f664509414112bc0d7f7cdb90c49a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41899 } } }
I20260927 02:37:34.508882 20156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 244f664509414112bc0d7f7cdb90c49a [sys.catalog]: This master's current role is: LEADER
I20260927 02:37:34.508857 20155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 244f664509414112bc0d7f7cdb90c49a [sys.catalog]: This master's current role is: LEADER
W20260927 02:37:34.512072 20170 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 244f664509414112bc0d7f7cdb90c49a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:37:34.512133 20170 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:37:34.512197 20164 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:37:34.512836 20164 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:37:34.517683 20164 catalog_manager.cc:1347] Generated new cluster ID: ab139ee226a64a0cbe50c97aa9af9d4f
I20260927 02:37:34.517738 20164 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:37:34.525611 20164 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:37:34.526719 20164 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:37:34.533545 20164 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 244f664509414112bc0d7f7cdb90c49a: Generated new TSK 0
I20260927 02:37:34.534179 20164 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:37:34.536603 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41899
--builtin_ntp_servers=127.0.98.20:34977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 02:37:34.638391 20173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:37:34.638602 20173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:37:34.638635 20173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:37:34.642455 20173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:37:34.642493 20173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:37:34.642542 20173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:37:34.647101 20173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:37:34.647882 20173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:37:34.649032 20173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:37:34.657370 20181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:34.657403 20178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:34.657388 20179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:34.657541 20173 server_base.cc:1034] running on GCE node
I20260927 02:37:34.657959 20173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:37:34.658515 20173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:37:34.659667 20173 hybrid_clock.cc:648] HybridClock initialized: now 1790476654659648 us; error 28 us; skew 500 ppm
I20260927 02:37:34.661326 20173 webserver.cc:460] Webserver started at http://127.0.98.1:44733/ using document root <none> and password file <none>
I20260927 02:37:34.661818 20173 fs_manager.cc:359] Metadata directory not provided
I20260927 02:37:34.661883 20173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:37:34.662058 20173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:37:34.663828 20173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-0/data/instance:
uuid: "b13df027a3b647f29879a3328a783e94"
format_stamp: "Formatted at 2026-09-27 02:37:34 on dist-test-slave-grlk"
server_key: "b6d1b418746f2cbf658f7318288f6a8b"
server_key_iv: "01242b2b6fc4a832cc8d7a6ed1adcc26"
server_key_version: "encryptionkey@0"
I20260927 02:37:34.664310 20173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b13df027a3b647f29879a3328a783e94"
format_stamp: "Formatted at 2026-09-27 02:37:34 on dist-test-slave-grlk"
server_key: "b6d1b418746f2cbf658f7318288f6a8b"
server_key_iv: "01242b2b6fc4a832cc8d7a6ed1adcc26"
server_key_version: "encryptionkey@0"
I20260927 02:37:34.667528 20173 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:37:34.669618 20187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:34.670619 20173 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:37:34.670723 20173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-0/wal
uuid: "b13df027a3b647f29879a3328a783e94"
format_stamp: "Formatted at 2026-09-27 02:37:34 on dist-test-slave-grlk"
server_key: "b6d1b418746f2cbf658f7318288f6a8b"
server_key_iv: "01242b2b6fc4a832cc8d7a6ed1adcc26"
server_key_version: "encryptionkey@0"
I20260927 02:37:34.670806 20173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:37:34.683573 20173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:37:34.684099 20173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:37:34.684243 20173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:37:34.684726 20173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:37:34.685554 20173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:37:34.685599 20173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:34.685647 20173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:37:34.685675 20173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:34.693212 20173 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34941
I20260927 02:37:34.693243 20300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34941 every 8 connection(s)
I20260927 02:37:34.693954 20173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:37:34.701560 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 20173
I20260927 02:37:34.701678 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:37:34.701676 20152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:34.695444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51279 (local address 127.0.98.62:41899)
0927 02:37:34.695588 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:34.695592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:34.696493 (+ 901us) server_negotiation.cc:407] Connection header received
0927 02:37:34.697742 (+ 1249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:34.697745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:34.697791 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:34.697852 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:34.698582 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:34.699097 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:34.700087 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:34.700303 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:34.700370 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:34.701219 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:34.701222 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:34.701224 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:37:34.701235 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:34.701259 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:37:34.701266 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:34.701318 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:34.701475 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 02:37:34.701535 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 02:37:34.701963 392 external_mini_cluster.cc:1428] Setting key 9cfb9e325e4506954fa5593202a540a1
I20260927 02:37:34.702399 20303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:34.695801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41899 (local address 127.0.98.1:51279)
0927 02:37:34.696372 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:34.696407 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:34.697565 (+ 1158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:34.697984 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:34.697990 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:34.698063 (+ 73us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:37:34.698483 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:34.698494 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:34.699179 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:34.699182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:34.700003 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:34.700011 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:34.700402 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:34.700404 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:34.700526 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:34.701012 (+ 486us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:37:34.701013 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:37:34.701047 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:37:34.701102 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:37:34.701325 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:37:34.701329 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:37:34.701332 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:37:34.701434 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 02:37:34.701668 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":101,"threads_started":1}
I20260927 02:37:34.703486 20301 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41899
I20260927 02:37:34.703670 20301 heartbeater.cc:464] Registering TS with master...
I20260927 02:37:34.703748 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41899
--builtin_ntp_servers=127.0.98.20:34977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 02:37:34.704119 20301 heartbeater.cc:511] Master 127.0.98.62:41899 requested a full tablet report, sending...
I20260927 02:37:34.705458 20112 ts_manager.cc:194] Registered new tserver with Master: b13df027a3b647f29879a3328a783e94 (127.0.98.1:34941)
I20260927 02:37:34.706519 20112 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:51279
W20260927 02:37:34.804505 20304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:37:34.804708 20304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:37:34.804767 20304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:37:34.808645 20304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:37:34.808686 20304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:37:34.808761 20304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:37:34.813246 20304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:37:34.814026 20304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:37:34.815352 20304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:37:34.823830 20310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:34.823868 20309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:34.823892 20312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:34.824074 20304 server_base.cc:1034] running on GCE node
I20260927 02:37:34.824404 20304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:37:34.824946 20304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:37:34.826105 20304 hybrid_clock.cc:648] HybridClock initialized: now 1790476654826093 us; error 27 us; skew 500 ppm
I20260927 02:37:34.827628 20304 webserver.cc:460] Webserver started at http://127.0.98.2:38335/ using document root <none> and password file <none>
I20260927 02:37:34.828116 20304 fs_manager.cc:359] Metadata directory not provided
I20260927 02:37:34.828176 20304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:37:34.828358 20304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:37:34.830193 20304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-1/data/instance:
uuid: "4c4f668b7b0d4811986fe80231710f1b"
format_stamp: "Formatted at 2026-09-27 02:37:34 on dist-test-slave-grlk"
server_key: "d8c57ae7712fbfe268b2328b9bef51f2"
server_key_iv: "21c00406c5192fb56011baae0c93294d"
server_key_version: "encryptionkey@0"
I20260927 02:37:34.830667 20304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4c4f668b7b0d4811986fe80231710f1b"
format_stamp: "Formatted at 2026-09-27 02:37:34 on dist-test-slave-grlk"
server_key: "d8c57ae7712fbfe268b2328b9bef51f2"
server_key_iv: "21c00406c5192fb56011baae0c93294d"
server_key_version: "encryptionkey@0"
I20260927 02:37:34.833869 20304 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:37:34.835945 20318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:34.836997 20304 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:37:34.837096 20304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-1/wal
uuid: "4c4f668b7b0d4811986fe80231710f1b"
format_stamp: "Formatted at 2026-09-27 02:37:34 on dist-test-slave-grlk"
server_key: "d8c57ae7712fbfe268b2328b9bef51f2"
server_key_iv: "21c00406c5192fb56011baae0c93294d"
server_key_version: "encryptionkey@0"
I20260927 02:37:34.837176 20304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:37:34.856644 20304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:37:34.857215 20304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:37:34.857357 20304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:37:34.857825 20304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:37:34.858608 20304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:37:34.858652 20304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:34.858698 20304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:37:34.858723 20304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:34.866351 20304 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37737
I20260927 02:37:34.866412 20431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37737 every 8 connection(s)
I20260927 02:37:34.867110 20304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:37:34.868556 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 20304
I20260927 02:37:34.868672 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:37:34.868975 392 external_mini_cluster.cc:1428] Setting key f2ef50cd5b0595c8429818a1b1c57bd8
I20260927 02:37:34.870918 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41899
--builtin_ntp_servers=127.0.98.20:34977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 02:37:34.876157 20152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:34.868668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39619 (local address 127.0.98.62:41899)
0927 02:37:34.868798 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:34.868803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:34.869881 (+ 1078us) server_negotiation.cc:407] Connection header received
0927 02:37:34.871308 (+ 1427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:34.871312 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:34.871371 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:34.871449 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:34.872290 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:34.872787 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:34.874373 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:34.874596 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:34.874670 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:34.875767 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:34.875769 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:34.875771 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:37:34.875782 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:34.875799 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:37:34.875806 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:34.875860 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:34.876009 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 02:37:34.876052 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 02:37:34.877226 20434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:34.869109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41899 (local address 127.0.98.2:39619)
0927 02:37:34.869740 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:34.869790 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:34.871125 (+ 1335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:34.871571 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:34.871577 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:34.871652 (+ 75us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:37:34.872159 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:34.872176 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:34.872897 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:34.872900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:34.874253 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:34.874275 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:34.874728 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:34.874732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:34.874929 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:34.875500 (+ 571us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:37:34.875502 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:37:34.875553 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:37:34.875628 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:37:34.875880 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:37:34.875888 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:37:34.875893 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:37:34.876015 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 02:37:34.876292 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":103,"threads_started":1}
I20260927 02:37:34.878224 20432 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41899
I20260927 02:37:34.878453 20432 heartbeater.cc:464] Registering TS with master...
I20260927 02:37:34.879055 20432 heartbeater.cc:511] Master 127.0.98.62:41899 requested a full tablet report, sending...
I20260927 02:37:34.880023 20112 ts_manager.cc:194] Registered new tserver with Master: 4c4f668b7b0d4811986fe80231710f1b (127.0.98.2:37737)
I20260927 02:37:34.880537 20112 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:39619
W20260927 02:37:34.975090 20435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:37:34.975315 20435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:37:34.975364 20435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:37:34.979326 20435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:37:34.979370 20435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:37:34.979439 20435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:37:34.983983 20435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:37:34.984874 20435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:37:34.986096 20435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:37:34.994599 20443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:34.994632 20441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:34.994598 20440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:34.994695 20435 server_base.cc:1034] running on GCE node
I20260927 02:37:34.995158 20435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:37:34.995679 20435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:37:34.996829 20435 hybrid_clock.cc:648] HybridClock initialized: now 1790476654996808 us; error 31 us; skew 500 ppm
I20260927 02:37:34.998445 20435 webserver.cc:460] Webserver started at http://127.0.98.3:33443/ using document root <none> and password file <none>
I20260927 02:37:34.998921 20435 fs_manager.cc:359] Metadata directory not provided
I20260927 02:37:34.998978 20435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:37:34.999154 20435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:37:35.001022 20435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-2/data/instance:
uuid: "7488d6de42974316ba6e561c6cc78538"
format_stamp: "Formatted at 2026-09-27 02:37:34 on dist-test-slave-grlk"
server_key: "08d22155439bd9d6d4f958882fd9d754"
server_key_iv: "b39eafacf11b78d6a709329b3a34305a"
server_key_version: "encryptionkey@0"
I20260927 02:37:35.001511 20435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7488d6de42974316ba6e561c6cc78538"
format_stamp: "Formatted at 2026-09-27 02:37:34 on dist-test-slave-grlk"
server_key: "08d22155439bd9d6d4f958882fd9d754"
server_key_iv: "b39eafacf11b78d6a709329b3a34305a"
server_key_version: "encryptionkey@0"
I20260927 02:37:35.004649 20435 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:37:35.006819 20449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:35.007908 20435 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:37:35.007994 20435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-2/wal
uuid: "7488d6de42974316ba6e561c6cc78538"
format_stamp: "Formatted at 2026-09-27 02:37:34 on dist-test-slave-grlk"
server_key: "08d22155439bd9d6d4f958882fd9d754"
server_key_iv: "b39eafacf11b78d6a709329b3a34305a"
server_key_version: "encryptionkey@0"
I20260927 02:37:35.008101 20435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:37:35.025012 20435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:37:35.025542 20435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:37:35.025676 20435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:37:35.026156 20435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:37:35.026961 20435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:37:35.027004 20435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:35.027052 20435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:37:35.027079 20435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:35.034956 20435 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44377
I20260927 02:37:35.035006 20562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44377 every 8 connection(s)
I20260927 02:37:35.035723 20435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:37:35.036402 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 20435
I20260927 02:37:35.036497 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:37:35.036722 392 external_mini_cluster.cc:1428] Setting key 22f80b7f69b1f3fcfed372a205f3fd7e
I20260927 02:37:35.043702 20152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:35.037328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42949 (local address 127.0.98.62:41899)
0927 02:37:35.037432 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:35.037437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:35.038575 (+ 1138us) server_negotiation.cc:407] Connection header received
0927 02:37:35.039857 (+ 1282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:35.039859 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:35.039901 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:35.039962 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:35.040695 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:35.041251 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:35.042272 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:35.042475 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:35.042537 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:35.043377 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:35.043379 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:35.043381 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:37:35.043391 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:35.043408 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:37:35.043415 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:35.043464 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:35.043558 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 02:37:35.043600 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 02:37:35.044399 20565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:35.037790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41899 (local address 127.0.98.3:42949)
0927 02:37:35.038457 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:35.038504 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:35.039691 (+ 1187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:35.040081 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:35.040087 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:35.040161 (+ 74us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:37:35.040588 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:35.040599 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:35.041343 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:35.041346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:35.042184 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:35.042192 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:35.042566 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:35.042569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:35.042696 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:35.043171 (+ 475us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:37:35.043173 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:37:35.043208 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:37:35.043261 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:37:35.043469 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:37:35.043473 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:37:35.043476 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:37:35.043551 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 02:37:35.043734 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":103,"threads_started":1}
I20260927 02:37:35.045293 20563 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41899
I20260927 02:37:35.045472 20563 heartbeater.cc:464] Registering TS with master...
I20260927 02:37:35.045895 20563 heartbeater.cc:511] Master 127.0.98.62:41899 requested a full tablet report, sending...
I20260927 02:37:35.046725 20112 ts_manager.cc:194] Registered new tserver with Master: 7488d6de42974316ba6e561c6cc78538 (127.0.98.3:44377)
I20260927 02:37:35.047214 20112 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:42949
I20260927 02:37:35.049890 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:37:35.054744 20152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:35.051509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53708 (local address 127.0.98.62:41899)
0927 02:37:35.051609 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:35.051613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:35.051761 (+ 148us) server_negotiation.cc:407] Connection header received
0927 02:37:35.051865 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:35.051868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:35.051907 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:35.051970 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:35.052206 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:35.052678 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:35.053499 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:35.053715 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:35.053794 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:35.054447 (+ 653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:35.054451 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:35.054453 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:37:35.054462 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:35.054478 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:37:35.054485 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:35.054535 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:35.054615 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 02:37:35.054654 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 02:37:35.056878 20112 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53708:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:37:35.059566 20112 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:37:35.071215 20574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:35.066561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34941 (local address 127.0.0.1:39294)
0927 02:37:35.066879 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:35.066923 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:35.067035 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:35.067574 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:35.067579 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:35.067598 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:37:35.067721 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:35.067728 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:35.069019 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:35.069021 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:35.069894 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:35.069903 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:35.070720 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:35.070723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:35.070834 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:35.070890 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 02:37:35.070968 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 02:37:35.071084 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":71,"threads_started":1}
I20260927 02:37:35.071914 20575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:35.066689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39294 (local address 127.0.98.1:34941)
0927 02:37:35.067119 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:35.067130 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:35.067152 (+ 22us) server_negotiation.cc:407] Connection header received
0927 02:37:35.067218 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:35.067223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:35.067425 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:35.067579 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:37:35.067819 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:35.068908 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:35.070009 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:35.070590 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:35.071536 (+ 946us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:35.071671 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:35.071708 (+ 37us) server_negotiation.cc:299] Negotiation successful
0927 02:37:35.071773 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":118,"threads_started":1}
I20260927 02:37:35.072227 20576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:35.066944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44377 (local address 127.0.0.1:34892)
0927 02:37:35.067412 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:35.067424 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:35.067493 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:35.068299 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:35.068302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:35.068318 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:37:35.068401 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:35.068406 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:35.069825 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:35.069828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:35.070930 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:35.070939 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:35.071892 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:35.071896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:35.071981 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:35.072003 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 02:37:35.072063 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 02:37:35.072116 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":82,"threads_started":1}
I20260927 02:37:35.072333 20578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:35.067502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34892 (local address 127.0.98.3:44377)
0927 02:37:35.067900 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:35.067911 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:35.067931 (+ 20us) server_negotiation.cc:407] Connection header received
0927 02:37:35.067990 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:35.067995 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:35.068159 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:35.068526 (+ 367us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:37:35.068630 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:35.069716 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:35.071058 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:35.071776 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:35.071891 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:35.072048 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:35.072095 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 02:37:35.072180 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":117,"threads_started":1}
I20260927 02:37:35.072540 20577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:35.067229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37737 (local address 127.0.0.1:47788)
0927 02:37:35.068283 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:35.068295 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:35.068393 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:35.068816 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:35.068819 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:35.068842 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:37:35.068924 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:35.068948 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:35.070183 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:35.070187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:35.071252 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:35.071260 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:35.072211 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:35.072215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:35.072292 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:35.072314 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 02:37:35.072372 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 02:37:35.072433 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1003,"thread_start_us":78,"threads_started":1}
I20260927 02:37:35.072618 20579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:35.067798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47788 (local address 127.0.98.2:37737)
0927 02:37:35.068102 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:35.068114 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:35.068487 (+ 373us) server_negotiation.cc:407] Connection header received
0927 02:37:35.068564 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:35.068569 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:35.068702 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:35.068820 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:37:35.069074 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:35.070090 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:35.071383 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:35.072103 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:35.072197 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:35.072363 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:35.072409 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 02:37:35.072492 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":121,"threads_started":1}
I20260927 02:37:35.075088 20235 tablet_service.cc:1467] Processing CreateTablet for tablet 13cd1b5589c34de1a455ff2266bb2797 (DEFAULT_TABLE table=test-table [id=f53f2f6871da4c0aa340a6cafcdf12c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:37:35.075372 20497 tablet_service.cc:1467] Processing CreateTablet for tablet 13cd1b5589c34de1a455ff2266bb2797 (DEFAULT_TABLE table=test-table [id=f53f2f6871da4c0aa340a6cafcdf12c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:37:35.075381 20366 tablet_service.cc:1467] Processing CreateTablet for tablet 13cd1b5589c34de1a455ff2266bb2797 (DEFAULT_TABLE table=test-table [id=f53f2f6871da4c0aa340a6cafcdf12c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:37:35.076025 20235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13cd1b5589c34de1a455ff2266bb2797. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:37:35.076301 20366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13cd1b5589c34de1a455ff2266bb2797. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:37:35.076301 20497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13cd1b5589c34de1a455ff2266bb2797. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:37:35.081869 20580 tablet_bootstrap.cc:492] T 13cd1b5589c34de1a455ff2266bb2797 P b13df027a3b647f29879a3328a783e94: Bootstrap starting.
I20260927 02:37:35.083283 20581 tablet_bootstrap.cc:492] T 13cd1b5589c34de1a455ff2266bb2797 P 7488d6de42974316ba6e561c6cc78538: Bootstrap starting.
I20260927 02:37:35.084414 20580 tablet_bootstrap.cc:654] T 13cd1b5589c34de1a455ff2266bb2797 P b13df027a3b647f29879a3328a783e94: Neither blocks nor log segments found. Creating new log.
I20260927 02:37:35.084790 20581 tablet_bootstrap.cc:654] T 13cd1b5589c34de1a455ff2266bb2797 P 7488d6de42974316ba6e561c6cc78538: Neither blocks nor log segments found. Creating new log.
I20260927 02:37:35.085119 20580 log.cc:824] T 13cd1b5589c34de1a455ff2266bb2797 P b13df027a3b647f29879a3328a783e94: Log is configured to *not* fsync() on all Append() calls
I20260927 02:37:35.085227 20582 tablet_bootstrap.cc:492] T 13cd1b5589c34de1a455ff2266bb2797 P 4c4f668b7b0d4811986fe80231710f1b: Bootstrap starting.
I20260927 02:37:35.085521 20581 log.cc:824] T 13cd1b5589c34de1a455ff2266bb2797 P 7488d6de42974316ba6e561c6cc78538: Log is configured to *not* fsync() on all Append() calls
I20260927 02:37:35.086762 20580 tablet_bootstrap.cc:492] T 13cd1b5589c34de1a455ff2266bb2797 P b13df027a3b647f29879a3328a783e94: No bootstrap required, opened a new log
I20260927 02:37:35.086938 20580 ts_tablet_manager.cc:1397] T 13cd1b5589c34de1a455ff2266bb2797 P b13df027a3b647f29879a3328a783e94: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 02:37:35.086988 20582 tablet_bootstrap.cc:654] T 13cd1b5589c34de1a455ff2266bb2797 P 4c4f668b7b0d4811986fe80231710f1b: Neither blocks nor log segments found. Creating new log.
I20260927 02:37:35.087276 20581 tablet_bootstrap.cc:492] T 13cd1b5589c34de1a455ff2266bb2797 P 7488d6de42974316ba6e561c6cc78538: No bootstrap required, opened a new log
I20260927 02:37:35.087441 20581 ts_tablet_manager.cc:1397] T 13cd1b5589c34de1a455ff2266bb2797 P 7488d6de42974316ba6e561c6cc78538: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260927 02:37:35.087723 20582 log.cc:824] T 13cd1b5589c34de1a455ff2266bb2797 P 4c4f668b7b0d4811986fe80231710f1b: Log is configured to *not* fsync() on all Append() calls
I20260927 02:37:35.090257 20580 raft_consensus.cc:348] T 13cd1b5589c34de1a455ff2266bb2797 P b13df027a3b647f29879a3328a783e94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13df027a3b647f29879a3328a783e94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34941 } } peers { permanent_uuid: "4c4f668b7b0d4811986fe80231710f1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37737 } } peers { permanent_uuid: "7488d6de42974316ba6e561c6cc78538" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44377 } }
I20260927 02:37:35.090451 20580 raft_consensus.cc:374] T 13cd1b5589c34de1a455ff2266bb2797 P b13df027a3b647f29879a3328a783e94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:37:35.090354 20581 raft_consensus.cc:348] T 13cd1b5589c34de1a455ff2266bb2797 P 7488d6de42974316ba6e561c6cc78538 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13df027a3b647f29879a3328a783e94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34941 } } peers { permanent_uuid: "4c4f668b7b0d4811986fe80231710f1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37737 } } peers { permanent_uuid: "7488d6de42974316ba6e561c6cc78538" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44377 } }
I20260927 02:37:35.090512 20580 raft_consensus.cc:729] T 13cd1b5589c34de1a455ff2266bb2797 P b13df027a3b647f29879a3328a783e94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b13df027a3b647f29879a3328a783e94, State: Initialized, Role: FOLLOWER
I20260927 02:37:35.090540 20581 raft_consensus.cc:374] T 13cd1b5589c34de1a455ff2266bb2797 P 7488d6de42974316ba6e561c6cc78538 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:37:35.090595 20581 raft_consensus.cc:729] T 13cd1b5589c34de1a455ff2266bb2797 P 7488d6de42974316ba6e561c6cc78538 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7488d6de42974316ba6e561c6cc78538, State: Initialized, Role: FOLLOWER
I20260927 02:37:35.090945 20580 consensus_queue.cc:260] T 13cd1b5589c34de1a455ff2266bb2797 P b13df027a3b647f29879a3328a783e94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13df027a3b647f29879a3328a783e94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34941 } } peers { permanent_uuid: "4c4f668b7b0d4811986fe80231710f1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37737 } } peers { permanent_uuid: "7488d6de42974316ba6e561c6cc78538" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44377 } }
I20260927 02:37:35.090986 20581 consensus_queue.cc:260] T 13cd1b5589c34de1a455ff2266bb2797 P 7488d6de42974316ba6e561c6cc78538 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13df027a3b647f29879a3328a783e94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34941 } } peers { permanent_uuid: "4c4f668b7b0d4811986fe80231710f1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37737 } } peers { permanent_uuid: "7488d6de42974316ba6e561c6cc78538" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44377 } }
I20260927 02:37:35.091506 20563 heartbeater.cc:503] Master 127.0.98.62:41899 was elected leader, sending a full tablet report...
I20260927 02:37:35.091817 20581 ts_tablet_manager.cc:1428] T 13cd1b5589c34de1a455ff2266bb2797 P 7488d6de42974316ba6e561c6cc78538: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20260927 02:37:35.091820 20580 ts_tablet_manager.cc:1428] T 13cd1b5589c34de1a455ff2266bb2797 P b13df027a3b647f29879a3328a783e94: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 02:37:35.092062 20582 tablet_bootstrap.cc:492] T 13cd1b5589c34de1a455ff2266bb2797 P 4c4f668b7b0d4811986fe80231710f1b: No bootstrap required, opened a new log
I20260927 02:37:35.092130 20301 heartbeater.cc:503] Master 127.0.98.62:41899 was elected leader, sending a full tablet report...
I20260927 02:37:35.092218 20582 ts_tablet_manager.cc:1397] T 13cd1b5589c34de1a455ff2266bb2797 P 4c4f668b7b0d4811986fe80231710f1b: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260927 02:37:35.095803 20582 raft_consensus.cc:348] T 13cd1b5589c34de1a455ff2266bb2797 P 4c4f668b7b0d4811986fe80231710f1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13df027a3b647f29879a3328a783e94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34941 } } peers { permanent_uuid: "4c4f668b7b0d4811986fe80231710f1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37737 } } peers { permanent_uuid: "7488d6de42974316ba6e561c6cc78538" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44377 } }
I20260927 02:37:35.096024 20582 raft_consensus.cc:374] T 13cd1b5589c34de1a455ff2266bb2797 P 4c4f668b7b0d4811986fe80231710f1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:37:35.096084 20582 raft_consensus.cc:729] T 13cd1b5589c34de1a455ff2266bb2797 P 4c4f668b7b0d4811986fe80231710f1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c4f668b7b0d4811986fe80231710f1b, State: Initialized, Role: FOLLOWER
I20260927 02:37:35.096410 20582 consensus_queue.cc:260] T 13cd1b5589c34de1a455ff2266bb2797 P 4c4f668b7b0d4811986fe80231710f1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13df027a3b647f29879a3328a783e94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34941 } } peers { permanent_uuid: "4c4f668b7b0d4811986fe80231710f1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37737 } } peers { permanent_uuid: "7488d6de42974316ba6e561c6cc78538" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44377 } }
I20260927 02:37:35.096907 20432 heartbeater.cc:503] Master 127.0.98.62:41899 was elected leader, sending a full tablet report...
I20260927 02:37:35.097110 20582 ts_tablet_manager.cc:1428] T 13cd1b5589c34de1a455ff2266bb2797 P 4c4f668b7b0d4811986fe80231710f1b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260927 02:37:35.117951 20433 tablet.cc:2369] T 13cd1b5589c34de1a455ff2266bb2797 P 4c4f668b7b0d4811986fe80231710f1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:37:35.189878 20588 raft_consensus.cc:482] T 13cd1b5589c34de1a455ff2266bb2797 P 4c4f668b7b0d4811986fe80231710f1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:37:35.190055 20588 raft_consensus.cc:504] T 13cd1b5589c34de1a455ff2266bb2797 P 4c4f668b7b0d4811986fe80231710f1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13df027a3b647f29879a3328a783e94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34941 } } peers { permanent_uuid: "4c4f668b7b0d4811986fe80231710f1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37737 } } peers { permanent_uuid: "7488d6de42974316ba6e561c6cc78538" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44377 } }
I20260927 02:37:35.191112 20588 leader_election.cc:288] T 13cd1b5589c34de1a455ff2266bb2797 P 4c4f668b7b0d4811986fe80231710f1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b13df027a3b647f29879a3328a783e94 (127.0.98.1:34941), 7488d6de42974316ba6e561c6cc78538 (127.0.98.3:44377)
I20260927 02:37:35.194661 20578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:35.191623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52071 (local address 127.0.98.3:44377)
0927 02:37:35.191729 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:35.191732 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:35.191743 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:37:35.191869 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:35.191872 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:35.191915 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:35.191990 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:37:35.192241 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:35.192748 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:35.193731 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:35.194113 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:35.194202 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:35.194255 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:35.194491 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 02:37:35.194553 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 02:37:35.194661 20575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:35.191371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44173 (local address 127.0.98.1:34941)
0927 02:37:35.191483 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:35.191488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:35.191592 (+ 104us) server_negotiation.cc:407] Connection header received
0927 02:37:35.191698 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:35.191701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:35.191752 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:35.191823 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:37:35.192091 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:35.192641 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:35.193709 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:35.194103 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:35.194189 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:35.194244 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:35.194476 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 02:37:35.194546 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 02:37:35.194662 20434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:35.191291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34941 (local address 127.0.98.2:44173)
0927 02:37:35.191481 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:35.191496 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:35.191582 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:35.191846 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:35.191848 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:35.191867 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:37:35.191980 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:35.191987 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:35.192740 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:35.192743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:35.193584 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:35.193593 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:35.194210 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:35.194213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:35.194342 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:35.194368 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 02:37:35.194436 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 02:37:35.194504 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":22}
I20260927 02:37:35.194680 20589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:35.191346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44377 (local address 127.0.98.2:52071)
0927 02:37:35.191656 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:35.191668 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:35.191754 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:35.192005 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:35.192007 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:35.192021 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:37:35.192146 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:35.192152 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:35.192845 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:35.192847 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:35.193619 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:35.193627 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:35.194229 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:35.194233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:35.194350 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:35.194396 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 02:37:35.194467 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 02:37:35.194532 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":33,"thread_start_us":107,"threads_started":1}
W20260927 02:37:35.194979 20302 tablet.cc:2369] T 13cd1b5589c34de1a455ff2266bb2797 P b13df027a3b647f29879a3328a783e94: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:37:35.195253 20255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13cd1b5589c34de1a455ff2266bb2797" candidate_uuid: "4c4f668b7b0d4811986fe80231710f1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b13df027a3b647f29879a3328a783e94" is_pre_election: true
I20260927 02:37:35.195272 20517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13cd1b5589c34de1a455ff2266bb2797" candidate_uuid: "4c4f668b7b0d4811986fe80231710f1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7488d6de42974316ba6e561c6cc78538" is_pre_election: true
I20260927 02:37:35.195526 20255 raft_consensus.cc:2454] T 13cd1b5589c34de1a455ff2266bb2797 P b13df027a3b647f29879a3328a783e94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c4f668b7b0d4811986fe80231710f1b in term 0.
I20260927 02:37:35.195530 20517 raft_consensus.cc:2454] T 13cd1b5589c34de1a455ff2266bb2797 P 7488d6de42974316ba6e561c6cc78538 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c4f668b7b0d4811986fe80231710f1b in term 0.
I20260927 02:37:35.195946 20321 leader_election.cc:302] T 13cd1b5589c34de1a455ff2266bb2797 P 4c4f668b7b0d4811986fe80231710f1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c4f668b7b0d4811986fe80231710f1b, b13df027a3b647f29879a3328a783e94; no voters:
I20260927 02:37:35.196187 20588 raft_consensus.cc:2781] T 13cd1b5589c34de1a455ff2266bb2797 P 4c4f668b7b0d4811986fe80231710f1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:37:35.196276 20588 raft_consensus.cc:482] T 13cd1b5589c34de1a455ff2266bb2797 P 4c4f668b7b0d4811986fe80231710f1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:37:35.196328 20588 raft_consensus.cc:3037] T 13cd1b5589c34de1a455ff2266bb2797 P 4c4f668b7b0d4811986fe80231710f1b [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:37:35.197244 20588 raft_consensus.cc:504] T 13cd1b5589c34de1a455ff2266bb2797 P 4c4f668b7b0d4811986fe80231710f1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13df027a3b647f29879a3328a783e94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34941 } } peers { permanent_uuid: "4c4f668b7b0d4811986fe80231710f1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37737 } } peers { permanent_uuid: "7488d6de42974316ba6e561c6cc78538" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44377 } }
I20260927 02:37:35.197516 20588 leader_election.cc:288] T 13cd1b5589c34de1a455ff2266bb2797 P 4c4f668b7b0d4811986fe80231710f1b [CANDIDATE]: Term 1 election: Requested vote from peers b13df027a3b647f29879a3328a783e94 (127.0.98.1:34941), 7488d6de42974316ba6e561c6cc78538 (127.0.98.3:44377)
I20260927 02:37:35.197825 20517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13cd1b5589c34de1a455ff2266bb2797" candidate_uuid: "4c4f668b7b0d4811986fe80231710f1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7488d6de42974316ba6e561c6cc78538"
I20260927 02:37:35.197829 20255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13cd1b5589c34de1a455ff2266bb2797" candidate_uuid: "4c4f668b7b0d4811986fe80231710f1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b13df027a3b647f29879a3328a783e94"
I20260927 02:37:35.197939 20255 raft_consensus.cc:3037] T 13cd1b5589c34de1a455ff2266bb2797 P b13df027a3b647f29879a3328a783e94 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:37:35.197939 20517 raft_consensus.cc:3037] T 13cd1b5589c34de1a455ff2266bb2797 P 7488d6de42974316ba6e561c6cc78538 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:37:35.199066 20255 raft_consensus.cc:2454] T 13cd1b5589c34de1a455ff2266bb2797 P b13df027a3b647f29879a3328a783e94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c4f668b7b0d4811986fe80231710f1b in term 1.
I20260927 02:37:35.199066 20517 raft_consensus.cc:2454] T 13cd1b5589c34de1a455ff2266bb2797 P 7488d6de42974316ba6e561c6cc78538 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c4f668b7b0d4811986fe80231710f1b in term 1.
I20260927 02:37:35.199348 20320 leader_election.cc:302] T 13cd1b5589c34de1a455ff2266bb2797 P 4c4f668b7b0d4811986fe80231710f1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c4f668b7b0d4811986fe80231710f1b, 7488d6de42974316ba6e561c6cc78538; no voters:
I20260927 02:37:35.199501 20588 raft_consensus.cc:2781] T 13cd1b5589c34de1a455ff2266bb2797 P 4c4f668b7b0d4811986fe80231710f1b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:37:35.199689 20588 raft_consensus.cc:686] T 13cd1b5589c34de1a455ff2266bb2797 P 4c4f668b7b0d4811986fe80231710f1b [term 1 LEADER]: Becoming Leader. State: Replica: 4c4f668b7b0d4811986fe80231710f1b, State: Running, Role: LEADER
I20260927 02:37:35.199963 20588 consensus_queue.cc:237] T 13cd1b5589c34de1a455ff2266bb2797 P 4c4f668b7b0d4811986fe80231710f1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13df027a3b647f29879a3328a783e94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34941 } } peers { permanent_uuid: "4c4f668b7b0d4811986fe80231710f1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37737 } } peers { permanent_uuid: "7488d6de42974316ba6e561c6cc78538" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44377 } }
I20260927 02:37:35.202733 20113 catalog_manager.cc:5495] T 13cd1b5589c34de1a455ff2266bb2797 P 4c4f668b7b0d4811986fe80231710f1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c4f668b7b0d4811986fe80231710f1b (127.0.98.2). New cstate: current_term: 1 leader_uuid: "4c4f668b7b0d4811986fe80231710f1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b13df027a3b647f29879a3328a783e94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c4f668b7b0d4811986fe80231710f1b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7488d6de42974316ba6e561c6cc78538" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44377 } health_report { overall_health: UNKNOWN } } }
I20260927 02:37:35.219139 20579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:35.216039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47794 (local address 127.0.98.2:37737)
0927 02:37:35.216129 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:35.216133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:35.216148 (+ 15us) server_negotiation.cc:407] Connection header received
0927 02:37:35.216302 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:35.216305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:35.216352 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:35.216431 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:37:35.216650 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:35.217124 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:35.217916 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:35.218111 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:35.218191 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:35.218460 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:37:35.218793 (+ 333us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:37:35.218899 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:35.218968 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 02:37:35.219011 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 02:37:35.223858 20517 raft_consensus.cc:1261] T 13cd1b5589c34de1a455ff2266bb2797 P 7488d6de42974316ba6e561c6cc78538 [term 1 FOLLOWER]: Refusing update from remote peer 4c4f668b7b0d4811986fe80231710f1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:37:35.223865 20255 raft_consensus.cc:1261] T 13cd1b5589c34de1a455ff2266bb2797 P b13df027a3b647f29879a3328a783e94 [term 1 FOLLOWER]: Refusing update from remote peer 4c4f668b7b0d4811986fe80231710f1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:37:35.224437 20588 consensus_queue.cc:1035] T 13cd1b5589c34de1a455ff2266bb2797 P 4c4f668b7b0d4811986fe80231710f1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7488d6de42974316ba6e561c6cc78538" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:37:35.224601 20590 consensus_queue.cc:1035] T 13cd1b5589c34de1a455ff2266bb2797 P 4c4f668b7b0d4811986fe80231710f1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b13df027a3b647f29879a3328a783e94" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:37:35.232273 20593 mvcc.cc:205] Tried to move back new op lower bound from 7333792379791212544 to 7333792379701145600. Current Snapshot: MvccSnapshot[applied={T|T < 7333792379791212544}]
I20260927 02:37:35.233210 20595 mvcc.cc:205] Tried to move back new op lower bound from 7333792379791212544 to 7333792379701145600. Current Snapshot: MvccSnapshot[applied={T|T < 7333792379791212544}]
I20260927 02:37:35.234777 20594 mvcc.cc:205] Tried to move back new op lower bound from 7333792379791212544 to 7333792379701145600. Current Snapshot: MvccSnapshot[applied={T|T < 7333792379791212544 or (T in {7333792379791212544})}]
I20260927 02:37:35.239599 20111 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:37:35.239718 20111 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:37:35.241745 20111 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 244f664509414112bc0d7f7cdb90c49a: Sending DeleteTablet for 3 replicas of tablet 13cd1b5589c34de1a455ff2266bb2797
I20260927 02:37:35.242425 20497 tablet_service.cc:1514] Processing DeleteTablet for tablet 13cd1b5589c34de1a455ff2266bb2797 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:37:35 UTC) from {username='slave'} at 127.0.0.1:34892
I20260927 02:37:35.242446 20235 tablet_service.cc:1514] Processing DeleteTablet for tablet 13cd1b5589c34de1a455ff2266bb2797 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:37:35 UTC) from {username='slave'} at 127.0.0.1:39294
I20260927 02:37:35.242549 20366 tablet_service.cc:1514] Processing DeleteTablet for tablet 13cd1b5589c34de1a455ff2266bb2797 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:37:35 UTC) from {username='slave'} at 127.0.0.1:47788
I20260927 02:37:35.242878 20603 tablet_replica.cc:330] T 13cd1b5589c34de1a455ff2266bb2797 P b13df027a3b647f29879a3328a783e94: stopping tablet replica
I20260927 02:37:35.243146 20603 raft_consensus.cc:2229] T 13cd1b5589c34de1a455ff2266bb2797 P b13df027a3b647f29879a3328a783e94 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:37:35.243445 20603 raft_consensus.cc:2258] T 13cd1b5589c34de1a455ff2266bb2797 P b13df027a3b647f29879a3328a783e94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:37:35.243531 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 20173
I20260927 02:37:35.243759 20604 tablet_replica.cc:330] T 13cd1b5589c34de1a455ff2266bb2797 P 4c4f668b7b0d4811986fe80231710f1b: stopping tablet replica
I20260927 02:37:35.244072 20604 raft_consensus.cc:2229] T 13cd1b5589c34de1a455ff2266bb2797 P 4c4f668b7b0d4811986fe80231710f1b [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:37:35.244374 20604 raft_consensus.cc:2258] T 13cd1b5589c34de1a455ff2266bb2797 P 4c4f668b7b0d4811986fe80231710f1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:37:35.245616 20604 ts_tablet_manager.cc:1905] T 13cd1b5589c34de1a455ff2266bb2797 P 4c4f668b7b0d4811986fe80231710f1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:37:35.248332 20604 ts_tablet_manager.cc:1918] T 13cd1b5589c34de1a455ff2266bb2797 P 4c4f668b7b0d4811986fe80231710f1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:37:35.248420 20604 log.cc:1196] T 13cd1b5589c34de1a455ff2266bb2797 P 4c4f668b7b0d4811986fe80231710f1b: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-1/wal/wals/13cd1b5589c34de1a455ff2266bb2797
I20260927 02:37:35.248648 20602 tablet_replica.cc:330] T 13cd1b5589c34de1a455ff2266bb2797 P 7488d6de42974316ba6e561c6cc78538: stopping tablet replica
I20260927 02:37:35.248673 20604 ts_tablet_manager.cc:1939] T 13cd1b5589c34de1a455ff2266bb2797 P 4c4f668b7b0d4811986fe80231710f1b: Deleting consensus metadata
I20260927 02:37:35.248988 20602 raft_consensus.cc:2229] T 13cd1b5589c34de1a455ff2266bb2797 P 7488d6de42974316ba6e561c6cc78538 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:37:35.249281 20602 raft_consensus.cc:2258] T 13cd1b5589c34de1a455ff2266bb2797 P 7488d6de42974316ba6e561c6cc78538 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:37:35.249655 20099 catalog_manager.cc:4841] TS 4c4f668b7b0d4811986fe80231710f1b (127.0.98.2:37737): tablet 13cd1b5589c34de1a455ff2266bb2797 (table test-table [id=f53f2f6871da4c0aa340a6cafcdf12c5]) successfully deleted
I20260927 02:37:35.250622 20602 ts_tablet_manager.cc:1905] T 13cd1b5589c34de1a455ff2266bb2797 P 7488d6de42974316ba6e561c6cc78538: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 02:37:35.250912 20099 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34941 (error 108)
I20260927 02:37:35.250965 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 20304
I20260927 02:37:35.252322 20577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:35.252079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34941 (local address 127.0.0.1:39298)
0927 02:37:35.252182 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:35.252245 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 02:37:35.252526 20099 catalog_manager.cc:4584] TS b13df027a3b647f29879a3328a783e94 (127.0.98.1:34941): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 13cd1b5589c34de1a455ff2266bb2797: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34941: connect: Connection refused (error 111)
I20260927 02:37:35.254261 20602 ts_tablet_manager.cc:1918] T 13cd1b5589c34de1a455ff2266bb2797 P 7488d6de42974316ba6e561c6cc78538: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:37:35.254371 20602 log.cc:1196] T 13cd1b5589c34de1a455ff2266bb2797 P 7488d6de42974316ba6e561c6cc78538: Deleting WAL directory at /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476410933508-392-0/minicluster-data/ts-2/wal/wals/13cd1b5589c34de1a455ff2266bb2797
I20260927 02:37:35.254657 20602 ts_tablet_manager.cc:1939] T 13cd1b5589c34de1a455ff2266bb2797 P 7488d6de42974316ba6e561c6cc78538: Deleting consensus metadata
I20260927 02:37:35.255647 20098 catalog_manager.cc:4841] TS 7488d6de42974316ba6e561c6cc78538 (127.0.98.3:44377): tablet 13cd1b5589c34de1a455ff2266bb2797 (table test-table [id=f53f2f6871da4c0aa340a6cafcdf12c5]) successfully deleted
I20260927 02:37:35.256968 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 20435
I20260927 02:37:35.261101 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 20082
2026-09-27T02:37:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (961 ms)
[----------] 6 tests from EncryptionPolicyTest (9481 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476410933508-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:37:35 dist-test-slave-grlk krb5kdc[20608](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:37:35 dist-test-slave-grlk krb5kdc[20608](info): set up 2 sockets
Sep 27 02:37:35 dist-test-slave-grlk krb5kdc[20608](info): commencing operation
krb5kdc: starting...
W20260927 02:37:36.200596 392 mini_kdc.cc:121] Time spent starting KDC: real 0.921s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476410933508-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476410933508-392-0/krb5kdc/test-user.keytab.
Sep 27 02:37:36 dist-test-slave-grlk krb5kdc[20608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:37:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:37:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476410933508-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476410933508-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:37:36.306308 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41965
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476410933508-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:37:36.408955 20621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:37:36.409189 20621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:37:36.409246 20621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:37:36.413118 20621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:37:36.413190 20621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:37:36.413216 20621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:37:36.413244 20621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:37:36.413272 20621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:37:36.418264 20621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41965
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476410933508-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:37:36.419130 20621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:37:36.420022 20621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:37:36.427620 20626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:36.427763 20621 server_base.cc:1034] running on GCE node
W20260927 02:37:36.427690 20629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:36.427690 20627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:36.428386 20621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:37:36.429370 20621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:37:36.430526 20621 hybrid_clock.cc:648] HybridClock initialized: now 1790476656430508 us; error 27 us; skew 500 ppm
Sep 27 02:37:36 dist-test-slave-grlk krb5kdc[20608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:37:36.437520 20621 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:37:36.438256 20621 webserver.cc:460] Webserver started at http://127.0.0.1:32913/ using document root <none> and password file <none>
I20260927 02:37:36.438715 20621 fs_manager.cc:359] Metadata directory not provided
I20260927 02:37:36.438783 20621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:37:36.438952 20621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:37:36.440762 20621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "9e12f56b33ca431b8297fc61c9d3a51f"
format_stamp: "Formatted at 2026-09-27 02:37:36 on dist-test-slave-grlk"
server_key: "2e945b759c4c10ca03240d76fc48f1c9"
server_key_iv: "b30c93d1508432975f447fc0bb6e586f"
server_key_version: "encryptionkey@0"
I20260927 02:37:36.441277 20621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "9e12f56b33ca431b8297fc61c9d3a51f"
format_stamp: "Formatted at 2026-09-27 02:37:36 on dist-test-slave-grlk"
server_key: "2e945b759c4c10ca03240d76fc48f1c9"
server_key_iv: "b30c93d1508432975f447fc0bb6e586f"
server_key_version: "encryptionkey@0"
I20260927 02:37:36.444445 20621 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 02:37:36.446535 20636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:36.447563 20621 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:37:36.447665 20621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "9e12f56b33ca431b8297fc61c9d3a51f"
format_stamp: "Formatted at 2026-09-27 02:37:36 on dist-test-slave-grlk"
server_key: "2e945b759c4c10ca03240d76fc48f1c9"
server_key_iv: "b30c93d1508432975f447fc0bb6e586f"
server_key_version: "encryptionkey@0"
I20260927 02:37:36.447758 20621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:37:36.459067 20621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:37:36.461444 20621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:37:36.461591 20621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:37:36.467552 20621 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41965
I20260927 02:37:36.467572 20688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41965 every 8 connection(s)
I20260927 02:37:36.468338 20621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:37:36.470723 20689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:37:36.471153 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 20621
I20260927 02:37:36.471297 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:37:36.471550 392 external_mini_cluster.cc:1428] Setting key 04be715fb6663ae0290e275cd662dbe3
I20260927 02:37:36.476682 20689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e12f56b33ca431b8297fc61c9d3a51f: Bootstrap starting.
I20260927 02:37:36.478914 20689 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e12f56b33ca431b8297fc61c9d3a51f: Neither blocks nor log segments found. Creating new log.
I20260927 02:37:36.479571 20689 log.cc:824] T 00000000000000000000000000000000 P 9e12f56b33ca431b8297fc61c9d3a51f: Log is configured to *not* fsync() on all Append() calls
Sep 27 02:37:36 dist-test-slave-grlk krb5kdc[20608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 02:37:36.481902 20689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e12f56b33ca431b8297fc61c9d3a51f: No bootstrap required, opened a new log
I20260927 02:37:36.485240 20692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:36.472742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60495 (local address 127.0.0.1:41965)
0927 02:37:36.473122 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:36.473138 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:36.473178 (+ 40us) server_negotiation.cc:407] Connection header received
0927 02:37:36.473943 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:36.473964 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:36.474346 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:36.474684 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:36.475282 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:36.476674 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:36.477554 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:36.478202 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:36.478347 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:36.480641 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:36.480647 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:36.480662 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:37:36.480688 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:36.482513 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:36.482947 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:36.482953 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:36.482958 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:36.483081 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:36.483332 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:36.483334 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:36.483336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:36.483787 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:36.484009 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:36.484168 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 02:37:36.484369 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":115,"threads_started":1}
I20260927 02:37:36.486123 20689 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e12f56b33ca431b8297fc61c9d3a51f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e12f56b33ca431b8297fc61c9d3a51f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41965 } }
I20260927 02:37:36.486367 20689 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e12f56b33ca431b8297fc61c9d3a51f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:37:36.486424 20689 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e12f56b33ca431b8297fc61c9d3a51f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e12f56b33ca431b8297fc61c9d3a51f, State: Initialized, Role: FOLLOWER
I20260927 02:37:36.486804 20689 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e12f56b33ca431b8297fc61c9d3a51f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e12f56b33ca431b8297fc61c9d3a51f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41965 } }
I20260927 02:37:36.486951 20689 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e12f56b33ca431b8297fc61c9d3a51f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:37:36.487006 20689 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e12f56b33ca431b8297fc61c9d3a51f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:37:36.487134 20689 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e12f56b33ca431b8297fc61c9d3a51f [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:37:36.487854 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:37:36.488027 20689 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e12f56b33ca431b8297fc61c9d3a51f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e12f56b33ca431b8297fc61c9d3a51f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41965 } }
I20260927 02:37:36.488319 20689 leader_election.cc:302] T 00000000000000000000000000000000 P 9e12f56b33ca431b8297fc61c9d3a51f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e12f56b33ca431b8297fc61c9d3a51f; no voters:
I20260927 02:37:36.488574 20689 leader_election.cc:288] T 00000000000000000000000000000000 P 9e12f56b33ca431b8297fc61c9d3a51f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:37:36.488673 20694 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e12f56b33ca431b8297fc61c9d3a51f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:37:36.488880 20694 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e12f56b33ca431b8297fc61c9d3a51f [term 1 LEADER]: Becoming Leader. State: Replica: 9e12f56b33ca431b8297fc61c9d3a51f, State: Running, Role: LEADER
I20260927 02:37:36.489233 20694 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e12f56b33ca431b8297fc61c9d3a51f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e12f56b33ca431b8297fc61c9d3a51f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41965 } }
I20260927 02:37:36.489619 20689 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e12f56b33ca431b8297fc61c9d3a51f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:37:36.490586 20695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e12f56b33ca431b8297fc61c9d3a51f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e12f56b33ca431b8297fc61c9d3a51f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e12f56b33ca431b8297fc61c9d3a51f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41965 } } }
I20260927 02:37:36.490800 20695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e12f56b33ca431b8297fc61c9d3a51f [sys.catalog]: This master's current role is: LEADER
I20260927 02:37:36.491150 20697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e12f56b33ca431b8297fc61c9d3a51f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e12f56b33ca431b8297fc61c9d3a51f. Latest consensus state: current_term: 1 leader_uuid: "9e12f56b33ca431b8297fc61c9d3a51f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e12f56b33ca431b8297fc61c9d3a51f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41965 } } }
I20260927 02:37:36.491307 20697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e12f56b33ca431b8297fc61c9d3a51f [sys.catalog]: This master's current role is: LEADER
W20260927 02:37:36.494107 20714 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9e12f56b33ca431b8297fc61c9d3a51f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:37:36.494181 20714 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:37:36.494236 20710 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:37:36.495028 20710 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:37:36.497608 20692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:36.490510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38025 (local address 127.0.0.1:41965)
0927 02:37:36.490606 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:36.490610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:36.491006 (+ 396us) server_negotiation.cc:407] Connection header received
0927 02:37:36.491179 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:36.491182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:36.491222 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:36.491270 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:36.492084 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:36.492560 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:36.493335 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:36.493548 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:36.493627 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:36.494853 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:36.494855 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:36.494857 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:37:36.494877 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:36.496078 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:36.496491 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:36.496494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:36.496495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:36.496558 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:36.496846 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:36.496848 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:36.496849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:36.497037 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:36.497158 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:36.497394 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 02:37:36.497472 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 02:37:36.500342 20710 catalog_manager.cc:1347] Generated new cluster ID: c2faa4b53ae9423aa55f52a945977b14
I20260927 02:37:36.500388 20710 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:37:36.515093 20710 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:37:36.515828 20710 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:37:36.524880 20710 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e12f56b33ca431b8297fc61c9d3a51f: Generated new TSK 0
I20260927 02:37:36.525444 20710 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:37:36.533628 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 20621
W20260927 02:37:36.538182 20606 connection.cc:535] client connection to 127.0.0.1:41965 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41965): Connection reset by peer (error 104)
2026-09-27T02:37:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1271 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476410933508-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:37:36 dist-test-slave-grlk krb5kdc[20720](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:37:36 dist-test-slave-grlk krb5kdc[20720](info): set up 2 sockets
Sep 27 02:37:36 dist-test-slave-grlk krb5kdc[20720](info): commencing operation
krb5kdc: starting...
W20260927 02:37:37.480643 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476410933508-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476410933508-392-0/krb5kdc/test-user.keytab.
Sep 27 02:37:37 dist-test-slave-grlk krb5kdc[20720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:37:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:37:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476410933508-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476410933508-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:37:37.585389 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40833
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476410933508-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:37:37.686456 20733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:37:37.686666 20733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:37:37.686702 20733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:37:37.690464 20733 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:37:37.690506 20733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:37:37.690524 20733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:37:37.690538 20733 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:37:37.690557 20733 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:37:37.695482 20733 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40833
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476410933508-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:37:37.696259 20733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:37:37.697094 20733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:37:37.704545 20741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:37.704540 20738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:37.704519 20739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:37.704598 20733 server_base.cc:1034] running on GCE node
I20260927 02:37:37.705209 20733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:37:37.706153 20733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:37:37.707311 20733 hybrid_clock.cc:648] HybridClock initialized: now 1790476657707290 us; error 33 us; skew 500 ppm
Sep 27 02:37:37 dist-test-slave-grlk krb5kdc[20720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:37:37.709673 20733 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:37:37.710450 20733 webserver.cc:460] Webserver started at http://127.0.0.1:43913/ using document root <none> and password file <none>
I20260927 02:37:37.710932 20733 fs_manager.cc:359] Metadata directory not provided
I20260927 02:37:37.710991 20733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:37:37.711154 20733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:37:37.713011 20733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "4ce009918e7843a68858d232b8b97873"
format_stamp: "Formatted at 2026-09-27 02:37:37 on dist-test-slave-grlk"
server_key: "1e9fa6cc78a403ecfff41bb8f14fbbde"
server_key_iv: "2eb825c95f6c718ccc79a7a68d50049d"
server_key_version: "encryptionkey@0"
I20260927 02:37:37.713539 20733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "4ce009918e7843a68858d232b8b97873"
format_stamp: "Formatted at 2026-09-27 02:37:37 on dist-test-slave-grlk"
server_key: "1e9fa6cc78a403ecfff41bb8f14fbbde"
server_key_iv: "2eb825c95f6c718ccc79a7a68d50049d"
server_key_version: "encryptionkey@0"
I20260927 02:37:37.716778 20733 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260927 02:37:37.718921 20748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:37.720014 20733 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:37:37.720122 20733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "4ce009918e7843a68858d232b8b97873"
format_stamp: "Formatted at 2026-09-27 02:37:37 on dist-test-slave-grlk"
server_key: "1e9fa6cc78a403ecfff41bb8f14fbbde"
server_key_iv: "2eb825c95f6c718ccc79a7a68d50049d"
server_key_version: "encryptionkey@0"
I20260927 02:37:37.720208 20733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:37:37.745788 20733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:37:37.748309 20733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:37:37.748458 20733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:37:37.754858 20733 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40833
I20260927 02:37:37.754887 20800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40833 every 8 connection(s)
I20260927 02:37:37.755686 20733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:37:37.758255 20801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:37:37.759848 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 20733
I20260927 02:37:37.759994 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:37:37.760283 392 external_mini_cluster.cc:1428] Setting key 34b58ce6528e29c6d5de3192db6591f4
I20260927 02:37:37.764163 20801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ce009918e7843a68858d232b8b97873: Bootstrap starting.
I20260927 02:37:37.766281 20801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ce009918e7843a68858d232b8b97873: Neither blocks nor log segments found. Creating new log.
I20260927 02:37:37.767019 20801 log.cc:824] T 00000000000000000000000000000000 P 4ce009918e7843a68858d232b8b97873: Log is configured to *not* fsync() on all Append() calls
I20260927 02:37:37.768632 20804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:37.761593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39509 (local address 127.0.0.1:40833)
0927 02:37:37.761986 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:37.762002 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:37.762035 (+ 33us) server_negotiation.cc:407] Connection header received
0927 02:37:37.762651 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:37.762668 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:37.762977 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:37.763259 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:37.763629 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:37.764671 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:37.765611 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:37.766053 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:37.766190 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:37.766984 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:37.766989 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:37.767001 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:37:37.767409 (+ 408us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:37.767468 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:37:37.767498 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:37.767630 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:37.767829 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 02:37:37.767954 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":122,"threads_started":1}
I20260927 02:37:37.768816 20801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ce009918e7843a68858d232b8b97873: No bootstrap required, opened a new log
I20260927 02:37:37.771281 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:37:37.772770 20801 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ce009918e7843a68858d232b8b97873 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ce009918e7843a68858d232b8b97873" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40833 } }
I20260927 02:37:37.773041 20801 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ce009918e7843a68858d232b8b97873 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:37:37.773101 20801 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ce009918e7843a68858d232b8b97873 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ce009918e7843a68858d232b8b97873, State: Initialized, Role: FOLLOWER
I20260927 02:37:37.773528 20801 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ce009918e7843a68858d232b8b97873 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ce009918e7843a68858d232b8b97873" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40833 } }
I20260927 02:37:37.773654 20801 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ce009918e7843a68858d232b8b97873 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:37:37.773753 20801 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ce009918e7843a68858d232b8b97873 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:37:37.773854 20801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ce009918e7843a68858d232b8b97873 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:37:37.774885 20801 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ce009918e7843a68858d232b8b97873 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ce009918e7843a68858d232b8b97873" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40833 } }
I20260927 02:37:37.775205 20801 leader_election.cc:302] T 00000000000000000000000000000000 P 4ce009918e7843a68858d232b8b97873 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ce009918e7843a68858d232b8b97873; no voters:
I20260927 02:37:37.775465 20801 leader_election.cc:288] T 00000000000000000000000000000000 P 4ce009918e7843a68858d232b8b97873 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:37:37.775554 20811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ce009918e7843a68858d232b8b97873 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:37:37.775748 20811 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ce009918e7843a68858d232b8b97873 [term 1 LEADER]: Becoming Leader. State: Replica: 4ce009918e7843a68858d232b8b97873, State: Running, Role: LEADER
I20260927 02:37:37.776111 20811 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ce009918e7843a68858d232b8b97873 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ce009918e7843a68858d232b8b97873" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40833 } }
I20260927 02:37:37.776530 20801 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ce009918e7843a68858d232b8b97873 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:37:37.777678 20813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ce009918e7843a68858d232b8b97873 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ce009918e7843a68858d232b8b97873" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ce009918e7843a68858d232b8b97873" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40833 } } }
I20260927 02:37:37.777843 20804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:37.773374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34261 (local address 127.0.0.1:40833)
0927 02:37:37.773478 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:37.773481 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:37.773670 (+ 189us) server_negotiation.cc:407] Connection header received
0927 02:37:37.773760 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:37.773764 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:37.773808 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:37.773890 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:37.774124 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:37.774862 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:37.775817 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:37.776128 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:37.776201 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:37.777139 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:37.777142 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:37.777145 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:37:37.777156 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:37.777183 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:37:37.777196 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:37.777290 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:37.777690 (+ 400us) server_negotiation.cc:299] Negotiation successful
0927 02:37:37.777723 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 02:37:37.778066 20813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ce009918e7843a68858d232b8b97873 [sys.catalog]: This master's current role is: LEADER
I20260927 02:37:37.777961 20812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ce009918e7843a68858d232b8b97873 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ce009918e7843a68858d232b8b97873. Latest consensus state: current_term: 1 leader_uuid: "4ce009918e7843a68858d232b8b97873" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ce009918e7843a68858d232b8b97873" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40833 } } }
I20260927 02:37:37.778334 20812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ce009918e7843a68858d232b8b97873 [sys.catalog]: This master's current role is: LEADER
W20260927 02:37:37.781571 20826 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4ce009918e7843a68858d232b8b97873: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:37:37.781682 20826 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:37:37.781801 20827 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:37:37.782696 20827 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:37:37.788554 20827 catalog_manager.cc:1347] Generated new cluster ID: bc8c10f0fb2b41fc90579b759ab24730
I20260927 02:37:37.788637 20827 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:37:37.815228 20827 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:37:37.816165 20827 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:37:37.820560 20827 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ce009918e7843a68858d232b8b97873: Generated new TSK 0
I20260927 02:37:37.821183 20827 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:37:37.825610 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 20733
W20260927 02:37:37.830472 20718 connection.cc:535] client connection to 127.0.0.1:40833 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40833): Connection reset by peer (error 104)
2026-09-27T02:37:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1292 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476410933508-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:37:37 dist-test-slave-grlk krb5kdc[20832](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:37:37 dist-test-slave-grlk krb5kdc[20832](info): set up 2 sockets
Sep 27 02:37:37 dist-test-slave-grlk krb5kdc[20832](info): commencing operation
krb5kdc: starting...
W20260927 02:37:38.780663 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476410933508-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476410933508-392-0/krb5kdc/test-user.keytab.
Sep 27 02:37:38 dist-test-slave-grlk krb5kdc[20832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:37:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:37:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476410933508-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476410933508-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:37:38.887501 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37509
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476410933508-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:37:38.990865 20845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:37:38.991072 20845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:37:38.991108 20845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:37:38.994899 20845 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:37:38.994944 20845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:37:38.994961 20845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:37:38.994975 20845 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:37:38.994989 20845 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:37:38.999868 20845 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37509
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476410933508-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:37:39.000664 20845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:37:39.001513 20845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:37:39.009022 20851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:39.009022 20850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:39.009143 20853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:39.009157 20845 server_base.cc:1034] running on GCE node
I20260927 02:37:39.009752 20845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:37:39.010701 20845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:37:39.011876 20845 hybrid_clock.cc:648] HybridClock initialized: now 1790476659011855 us; error 34 us; skew 500 ppm
Sep 27 02:37:39 dist-test-slave-grlk krb5kdc[20832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476659, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:37:39.014353 20845 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:37:39.015144 20845 webserver.cc:460] Webserver started at http://127.0.0.1:46173/ using document root <none> and password file <none>
I20260927 02:37:39.015604 20845 fs_manager.cc:359] Metadata directory not provided
I20260927 02:37:39.015654 20845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:37:39.015818 20845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:37:39.017678 20845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "fcecd23390064bbe9ce12332c78da79a"
format_stamp: "Formatted at 2026-09-27 02:37:39 on dist-test-slave-grlk"
server_key: "fceef24dca367e6715b75e31915dc7e1"
server_key_iv: "7898e8bd59edca8fbd315c6f597e719e"
server_key_version: "encryptionkey@0"
I20260927 02:37:39.018149 20845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "fcecd23390064bbe9ce12332c78da79a"
format_stamp: "Formatted at 2026-09-27 02:37:39 on dist-test-slave-grlk"
server_key: "fceef24dca367e6715b75e31915dc7e1"
server_key_iv: "7898e8bd59edca8fbd315c6f597e719e"
server_key_version: "encryptionkey@0"
I20260927 02:37:39.021421 20845 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:37:39.023515 20860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:39.024595 20845 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:37:39.024693 20845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "fcecd23390064bbe9ce12332c78da79a"
format_stamp: "Formatted at 2026-09-27 02:37:39 on dist-test-slave-grlk"
server_key: "fceef24dca367e6715b75e31915dc7e1"
server_key_iv: "7898e8bd59edca8fbd315c6f597e719e"
server_key_version: "encryptionkey@0"
I20260927 02:37:39.024780 20845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:37:39.029003 20845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:37:39.031601 20845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:37:39.031756 20845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:37:39.038607 20845 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37509
I20260927 02:37:39.038651 20912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37509 every 8 connection(s)
I20260927 02:37:39.039518 20845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:37:39.042099 20913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:37:39.042161 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 20845
I20260927 02:37:39.042302 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:37:39.042593 392 external_mini_cluster.cc:1428] Setting key d6c4d867e01c544d3f9d741bbb77edcb
I20260927 02:37:39.047478 20916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:39.043885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38271 (local address 127.0.0.1:37509)
0927 02:37:39.044303 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:39.044318 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:39.044350 (+ 32us) server_negotiation.cc:407] Connection header received
0927 02:37:39.045123 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:39.045140 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:39.045402 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:39.045684 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:39.046377 (+ 693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:39.046382 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:39.046393 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:37:39.046402 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:39.046451 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:37:39.046470 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:39.046597 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:39.046739 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 02:37:39.046879 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":132,"threads_started":1}
I20260927 02:37:39.048115 20913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcecd23390064bbe9ce12332c78da79a: Bootstrap starting.
I20260927 02:37:39.049798 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:37:39.050906 20913 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcecd23390064bbe9ce12332c78da79a: Neither blocks nor log segments found. Creating new log.
I20260927 02:37:39.051654 20913 log.cc:824] T 00000000000000000000000000000000 P fcecd23390064bbe9ce12332c78da79a: Log is configured to *not* fsync() on all Append() calls
I20260927 02:37:39.052919 20916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:39.051551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35437 (local address 127.0.0.1:37509)
0927 02:37:39.051644 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:39.051648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:39.051826 (+ 178us) server_negotiation.cc:407] Connection header received
0927 02:37:39.051940 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:39.051943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:39.051971 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:39.052030 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:39.052605 (+ 575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:39.052607 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:39.052610 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:37:39.052619 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:39.052640 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:37:39.052649 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:39.052713 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:39.052800 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 02:37:39.052825 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 02:37:39.053517 20913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcecd23390064bbe9ce12332c78da79a: No bootstrap required, opened a new log
I20260927 02:37:39.056843 20913 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcecd23390064bbe9ce12332c78da79a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcecd23390064bbe9ce12332c78da79a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37509 } }
I20260927 02:37:39.057070 20913 raft_consensus.cc:374] T 00000000000000000000000000000000 P fcecd23390064bbe9ce12332c78da79a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:37:39.057124 20913 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcecd23390064bbe9ce12332c78da79a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcecd23390064bbe9ce12332c78da79a, State: Initialized, Role: FOLLOWER
I20260927 02:37:39.057502 20913 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcecd23390064bbe9ce12332c78da79a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcecd23390064bbe9ce12332c78da79a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37509 } }
I20260927 02:37:39.057622 20913 raft_consensus.cc:388] T 00000000000000000000000000000000 P fcecd23390064bbe9ce12332c78da79a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:37:39.057683 20913 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcecd23390064bbe9ce12332c78da79a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:37:39.057772 20913 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcecd23390064bbe9ce12332c78da79a [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:37:39.058523 20913 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcecd23390064bbe9ce12332c78da79a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcecd23390064bbe9ce12332c78da79a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37509 } }
I20260927 02:37:39.058818 20913 leader_election.cc:302] T 00000000000000000000000000000000 P fcecd23390064bbe9ce12332c78da79a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fcecd23390064bbe9ce12332c78da79a; no voters:
I20260927 02:37:39.059067 20913 leader_election.cc:288] T 00000000000000000000000000000000 P fcecd23390064bbe9ce12332c78da79a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:37:39.059145 20923 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fcecd23390064bbe9ce12332c78da79a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:37:39.059331 20923 raft_consensus.cc:686] T 00000000000000000000000000000000 P fcecd23390064bbe9ce12332c78da79a [term 1 LEADER]: Becoming Leader. State: Replica: fcecd23390064bbe9ce12332c78da79a, State: Running, Role: LEADER
I20260927 02:37:39.059644 20923 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcecd23390064bbe9ce12332c78da79a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcecd23390064bbe9ce12332c78da79a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37509 } }
I20260927 02:37:39.060117 20913 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcecd23390064bbe9ce12332c78da79a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:37:39.061270 20924 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcecd23390064bbe9ce12332c78da79a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fcecd23390064bbe9ce12332c78da79a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcecd23390064bbe9ce12332c78da79a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37509 } } }
I20260927 02:37:39.061331 20925 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcecd23390064bbe9ce12332c78da79a [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcecd23390064bbe9ce12332c78da79a. Latest consensus state: current_term: 1 leader_uuid: "fcecd23390064bbe9ce12332c78da79a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcecd23390064bbe9ce12332c78da79a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37509 } } }
I20260927 02:37:39.061429 20925 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcecd23390064bbe9ce12332c78da79a [sys.catalog]: This master's current role is: LEADER
I20260927 02:37:39.061381 20924 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcecd23390064bbe9ce12332c78da79a [sys.catalog]: This master's current role is: LEADER
I20260927 02:37:39.062243 20933 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:37:39.064649 20933 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:37:39.069346 20933 catalog_manager.cc:1347] Generated new cluster ID: 7fe49feef8974f03a81c54592035b0cb
I20260927 02:37:39.069401 20933 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:37:39.100225 20933 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:37:39.100539 20933 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:37:39.104601 20933 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fcecd23390064bbe9ce12332c78da79a: Generated new TSK 0
I20260927 02:37:39.105218 20933 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:37:39.111878 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 20845
2026-09-27T02:37:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1287 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476410933508-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:37:39 dist-test-slave-grlk krb5kdc[20944](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:37:39 dist-test-slave-grlk krb5kdc[20944](info): set up 2 sockets
Sep 27 02:37:39 dist-test-slave-grlk krb5kdc[20944](info): commencing operation
krb5kdc: starting...
W20260927 02:37:40.056653 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476410933508-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476410933508-392-0/krb5kdc/test-user.keytab.
Sep 27 02:37:40 dist-test-slave-grlk krb5kdc[20944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:37:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:37:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476410933508-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476410933508-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:37:40.163009 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41211
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476410933508-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:37:40.264304 20957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:37:40.264535 20957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:37:40.264588 20957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:37:40.268462 20957 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:37:40.268524 20957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:37:40.268549 20957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:37:40.268569 20957 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:37:40.268589 20957 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:37:40.273664 20957 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476410933508-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:37:40.274595 20957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:37:40.275519 20957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:37:40.283252 20963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:40.283357 20957 server_base.cc:1034] running on GCE node
W20260927 02:37:40.283217 20962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:40.283236 20965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:40.283965 20957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:37:40.284899 20957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:37:40.286068 20957 hybrid_clock.cc:648] HybridClock initialized: now 1790476660286053 us; error 27 us; skew 500 ppm
Sep 27 02:37:40 dist-test-slave-grlk krb5kdc[20944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:37:40.288561 20957 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:37:40.289433 20957 webserver.cc:460] Webserver started at http://127.0.0.1:44611/ using document root <none> and password file <none>
I20260927 02:37:40.289971 20957 fs_manager.cc:359] Metadata directory not provided
I20260927 02:37:40.290042 20957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:37:40.290202 20957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:37:40.292052 20957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "6666c307f13647af86f45e9e570cbf50"
format_stamp: "Formatted at 2026-09-27 02:37:40 on dist-test-slave-grlk"
server_key: "e25ce397df4f7ba5968e42f33f63bcc7"
server_key_iv: "8990377dd685c9f9cd4fb429e83d472f"
server_key_version: "encryptionkey@0"
I20260927 02:37:40.292521 20957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "6666c307f13647af86f45e9e570cbf50"
format_stamp: "Formatted at 2026-09-27 02:37:40 on dist-test-slave-grlk"
server_key: "e25ce397df4f7ba5968e42f33f63bcc7"
server_key_iv: "8990377dd685c9f9cd4fb429e83d472f"
server_key_version: "encryptionkey@0"
I20260927 02:37:40.296166 20957 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:37:40.298511 20972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:40.299655 20957 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:37:40.299750 20957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "6666c307f13647af86f45e9e570cbf50"
format_stamp: "Formatted at 2026-09-27 02:37:40 on dist-test-slave-grlk"
server_key: "e25ce397df4f7ba5968e42f33f63bcc7"
server_key_iv: "8990377dd685c9f9cd4fb429e83d472f"
server_key_version: "encryptionkey@0"
I20260927 02:37:40.299835 20957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:37:40.326640 20957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:37:40.329557 20957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:37:40.329692 20957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:37:40.336009 20957 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41211
I20260927 02:37:40.336030 21024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41211 every 8 connection(s)
I20260927 02:37:40.336825 20957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:37:40.337651 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 20957
I20260927 02:37:40.337761 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:37:40.338001 392 external_mini_cluster.cc:1428] Setting key c876c9bdf565518fbca468d9154996ed
I20260927 02:37:40.339645 21025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:37:40.346240 21025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6666c307f13647af86f45e9e570cbf50: Bootstrap starting.
Sep 27 02:37:40 dist-test-slave-grlk krb5kdc[20944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 02:37:40.348555 21025 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6666c307f13647af86f45e9e570cbf50: Neither blocks nor log segments found. Creating new log.
I20260927 02:37:40.349310 21025 log.cc:824] T 00000000000000000000000000000000 P 6666c307f13647af86f45e9e570cbf50: Log is configured to *not* fsync() on all Append() calls
I20260927 02:37:40.351155 21025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6666c307f13647af86f45e9e570cbf50: No bootstrap required, opened a new log
I20260927 02:37:40.351588 21028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:40.339216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60497 (local address 127.0.0.1:41211)
0927 02:37:40.339578 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:40.339592 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:40.339625 (+ 33us) server_negotiation.cc:407] Connection header received
0927 02:37:40.340242 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:40.340257 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:40.340561 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:40.340814 (+ 253us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:40.341643 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:40.342646 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:40.343755 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:40.344223 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:40.344325 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:40.347233 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:40.347238 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:40.347251 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:37:40.347278 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:40.349058 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:40.349447 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:40.349453 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:40.349457 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:40.349563 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:40.349803 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:40.349806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:40.349808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:40.350182 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:40.350348 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:40.350625 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 02:37:40.350892 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":242,"thread_start_us":111,"threads_started":1}
I20260927 02:37:40.354840 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:37:40.355147 21025 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6666c307f13647af86f45e9e570cbf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6666c307f13647af86f45e9e570cbf50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41211 } }
I20260927 02:37:40.355376 21025 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6666c307f13647af86f45e9e570cbf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:37:40.355432 21025 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6666c307f13647af86f45e9e570cbf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6666c307f13647af86f45e9e570cbf50, State: Initialized, Role: FOLLOWER
I20260927 02:37:40.355836 21025 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6666c307f13647af86f45e9e570cbf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6666c307f13647af86f45e9e570cbf50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41211 } }
I20260927 02:37:40.355963 21025 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6666c307f13647af86f45e9e570cbf50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:37:40.356029 21025 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6666c307f13647af86f45e9e570cbf50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:37:40.356117 21025 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6666c307f13647af86f45e9e570cbf50 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:37:40.357235 21025 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6666c307f13647af86f45e9e570cbf50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6666c307f13647af86f45e9e570cbf50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41211 } }
I20260927 02:37:40.357544 21025 leader_election.cc:302] T 00000000000000000000000000000000 P 6666c307f13647af86f45e9e570cbf50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6666c307f13647af86f45e9e570cbf50; no voters:
I20260927 02:37:40.357816 21025 leader_election.cc:288] T 00000000000000000000000000000000 P 6666c307f13647af86f45e9e570cbf50 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:37:40.357928 21035 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6666c307f13647af86f45e9e570cbf50 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:37:40.358108 21035 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6666c307f13647af86f45e9e570cbf50 [term 1 LEADER]: Becoming Leader. State: Replica: 6666c307f13647af86f45e9e570cbf50, State: Running, Role: LEADER
I20260927 02:37:40.358446 21035 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6666c307f13647af86f45e9e570cbf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6666c307f13647af86f45e9e570cbf50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41211 } }
I20260927 02:37:40.359048 21025 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6666c307f13647af86f45e9e570cbf50 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:37:40.359583 21035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6666c307f13647af86f45e9e570cbf50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6666c307f13647af86f45e9e570cbf50. Latest consensus state: current_term: 1 leader_uuid: "6666c307f13647af86f45e9e570cbf50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6666c307f13647af86f45e9e570cbf50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41211 } } }
I20260927 02:37:40.359692 21035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6666c307f13647af86f45e9e570cbf50 [sys.catalog]: This master's current role is: LEADER
I20260927 02:37:40.359936 21037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6666c307f13647af86f45e9e570cbf50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6666c307f13647af86f45e9e570cbf50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6666c307f13647af86f45e9e570cbf50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41211 } } }
I20260927 02:37:40.360035 21037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6666c307f13647af86f45e9e570cbf50 [sys.catalog]: This master's current role is: LEADER
I20260927 02:37:40.360054 21041 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:37:40.363969 21041 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:37:40.365451 21028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:40.356853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59753 (local address 127.0.0.1:41211)
0927 02:37:40.357004 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:40.357009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:40.357174 (+ 165us) server_negotiation.cc:407] Connection header received
0927 02:37:40.357290 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:40.357294 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:40.357344 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:40.357413 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:40.358200 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:40.358959 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:40.360026 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:40.360343 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:40.360417 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:40.362082 (+ 1665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:40.362086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:40.362089 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:37:40.362111 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:40.363626 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:40.364077 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:40.364080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:40.364082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:40.364167 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:40.364610 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:40.364614 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:40.364615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:40.364786 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:40.364898 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:40.365200 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 02:37:40.365306 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 02:37:40.369448 21041 catalog_manager.cc:1347] Generated new cluster ID: 195dce28187f4b04816675faf952a390
I20260927 02:37:40.369494 21041 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:37:40.392829 21041 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:37:40.393800 21041 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:37:40.400410 21041 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6666c307f13647af86f45e9e570cbf50: Generated new TSK 0
I20260927 02:37:40.401198 21041 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:37:40.414222 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 20957
2026-09-27T02:37:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1301 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476410933508-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:37:40 dist-test-slave-grlk krb5kdc[21056](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:37:40 dist-test-slave-grlk krb5kdc[21056](info): set up 2 sockets
Sep 27 02:37:40 dist-test-slave-grlk krb5kdc[21056](info): commencing operation
krb5kdc: starting...
W20260927 02:37:41.364975 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476410933508-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476410933508-392-0/krb5kdc/test-user.keytab.
Sep 27 02:37:41 dist-test-slave-grlk krb5kdc[21056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:37:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:37:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476410933508-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476410933508-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:37:41.472002 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41149
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476410933508-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:37:41.573437 21069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:37:41.573657 21069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:37:41.573695 21069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:37:41.577561 21069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:37:41.577610 21069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:37:41.577628 21069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:37:41.577642 21069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:37:41.577656 21069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:37:41.582604 21069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476410933508-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:37:41.583460 21069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:37:41.584314 21069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:37:41.592061 21074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:41.592061 21075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:41.592087 21077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:41.592140 21069 server_base.cc:1034] running on GCE node
I20260927 02:37:41.592720 21069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:37:41.593751 21069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:37:41.594894 21069 hybrid_clock.cc:648] HybridClock initialized: now 1790476661594875 us; error 31 us; skew 500 ppm
Sep 27 02:37:41 dist-test-slave-grlk krb5kdc[21056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476661, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:37:41.600322 21069 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:37:41.601178 21069 webserver.cc:460] Webserver started at http://127.0.0.1:40117/ using document root <none> and password file <none>
I20260927 02:37:41.601675 21069 fs_manager.cc:359] Metadata directory not provided
I20260927 02:37:41.601732 21069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:37:41.601894 21069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:37:41.603765 21069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "f6f09d29a6c646159f3d95e8d452587f"
format_stamp: "Formatted at 2026-09-27 02:37:41 on dist-test-slave-grlk"
server_key: "5d72e0d34951026e6f0069219ffc0a85"
server_key_iv: "9ec7e9ec2ab2e2e86778b71e6a1b2c0a"
server_key_version: "encryptionkey@0"
I20260927 02:37:41.604233 21069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "f6f09d29a6c646159f3d95e8d452587f"
format_stamp: "Formatted at 2026-09-27 02:37:41 on dist-test-slave-grlk"
server_key: "5d72e0d34951026e6f0069219ffc0a85"
server_key_iv: "9ec7e9ec2ab2e2e86778b71e6a1b2c0a"
server_key_version: "encryptionkey@0"
I20260927 02:37:41.607752 21069 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 02:37:41.609844 21084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:41.610888 21069 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 02:37:41.610980 21069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "f6f09d29a6c646159f3d95e8d452587f"
format_stamp: "Formatted at 2026-09-27 02:37:41 on dist-test-slave-grlk"
server_key: "5d72e0d34951026e6f0069219ffc0a85"
server_key_iv: "9ec7e9ec2ab2e2e86778b71e6a1b2c0a"
server_key_version: "encryptionkey@0"
I20260927 02:37:41.611058 21069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:37:41.627154 21069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:37:41.629962 21069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:37:41.630096 21069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:37:41.636262 21069 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41149
I20260927 02:37:41.636269 21136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41149 every 8 connection(s)
I20260927 02:37:41.637097 21069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:37:41.639513 21137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:37:41.644363 21137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6f09d29a6c646159f3d95e8d452587f: Bootstrap starting.
I20260927 02:37:41.646659 21137 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6f09d29a6c646159f3d95e8d452587f: Neither blocks nor log segments found. Creating new log.
I20260927 02:37:41.646756 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 21069
I20260927 02:37:41.646922 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:37:41.647174 392 external_mini_cluster.cc:1428] Setting key 7758caf9637b2844452a430bb5d620af
I20260927 02:37:41.647538 21137 log.cc:824] T 00000000000000000000000000000000 P f6f09d29a6c646159f3d95e8d452587f: Log is configured to *not* fsync() on all Append() calls
I20260927 02:37:41.649706 21137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6f09d29a6c646159f3d95e8d452587f: No bootstrap required, opened a new log
I20260927 02:37:41.654209 21137 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6f09d29a6c646159f3d95e8d452587f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f09d29a6c646159f3d95e8d452587f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41149 } }
I20260927 02:37:41.654497 21137 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6f09d29a6c646159f3d95e8d452587f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:37:41.654565 21137 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6f09d29a6c646159f3d95e8d452587f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6f09d29a6c646159f3d95e8d452587f, State: Initialized, Role: FOLLOWER
I20260927 02:37:41.655014 21137 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6f09d29a6c646159f3d95e8d452587f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f09d29a6c646159f3d95e8d452587f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41149 } }
I20260927 02:37:41.655189 21137 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6f09d29a6c646159f3d95e8d452587f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:37:41.655261 21137 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6f09d29a6c646159f3d95e8d452587f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:37:41.655359 21137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6f09d29a6c646159f3d95e8d452587f [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:37:41.656457 21137 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6f09d29a6c646159f3d95e8d452587f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f09d29a6c646159f3d95e8d452587f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41149 } }
I20260927 02:37:41.656781 21141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:41.648227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43445 (local address 127.0.0.1:41149)
0927 02:37:41.648645 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:41.648676 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:41.648720 (+ 44us) server_negotiation.cc:407] Connection header received
0927 02:37:41.649601 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:41.649623 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:41.650032 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:41.650382 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:41.650784 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:41.652227 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:41.653286 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:41.653987 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:41.654160 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:41.654854 (+ 694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:41.654861 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:41.654875 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:37:41.655350 (+ 475us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:41.655422 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:37:41.655456 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:41.655557 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:41.655674 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 02:37:41.655867 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":118,"threads_started":1}
I20260927 02:37:41.656803 21137 leader_election.cc:302] T 00000000000000000000000000000000 P f6f09d29a6c646159f3d95e8d452587f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6f09d29a6c646159f3d95e8d452587f; no voters:
I20260927 02:37:41.657120 21137 leader_election.cc:288] T 00000000000000000000000000000000 P f6f09d29a6c646159f3d95e8d452587f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:37:41.657202 21142 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6f09d29a6c646159f3d95e8d452587f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:37:41.657413 21142 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6f09d29a6c646159f3d95e8d452587f [term 1 LEADER]: Becoming Leader. State: Replica: f6f09d29a6c646159f3d95e8d452587f, State: Running, Role: LEADER
I20260927 02:37:41.657729 21142 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6f09d29a6c646159f3d95e8d452587f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f09d29a6c646159f3d95e8d452587f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41149 } }
I20260927 02:37:41.658339 21137 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6f09d29a6c646159f3d95e8d452587f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:37:41.658690 21142 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6f09d29a6c646159f3d95e8d452587f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6f09d29a6c646159f3d95e8d452587f. Latest consensus state: current_term: 1 leader_uuid: "f6f09d29a6c646159f3d95e8d452587f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f09d29a6c646159f3d95e8d452587f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41149 } } }
I20260927 02:37:41.658798 21142 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6f09d29a6c646159f3d95e8d452587f [sys.catalog]: This master's current role is: LEADER
I20260927 02:37:41.659582 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:37:41.660564 21146 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:37:41.660789 21143 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6f09d29a6c646159f3d95e8d452587f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6f09d29a6c646159f3d95e8d452587f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f09d29a6c646159f3d95e8d452587f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41149 } } }
I20260927 02:37:41.661264 21143 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6f09d29a6c646159f3d95e8d452587f [sys.catalog]: This master's current role is: LEADER
I20260927 02:37:41.663509 21146 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:37:41.666033 21141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:41.661790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50187 (local address 127.0.0.1:41149)
0927 02:37:41.661901 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:41.661905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:41.662064 (+ 159us) server_negotiation.cc:407] Connection header received
0927 02:37:41.662219 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:41.662222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:41.662262 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:41.662305 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:41.662661 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:41.663361 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:41.664420 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:41.664719 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:41.664789 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:41.665663 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:41.665667 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:41.665670 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:37:41.665681 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:41.665705 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:37:41.665714 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:41.665770 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:41.665866 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 02:37:41.665910 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 02:37:41.669070 21146 catalog_manager.cc:1347] Generated new cluster ID: 4830576f37884250adf81b2c134634f5
I20260927 02:37:41.669126 21146 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:37:41.677223 21146 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:37:41.678035 21146 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:37:41.681819 21146 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6f09d29a6c646159f3d95e8d452587f: Generated new TSK 0
I20260927 02:37:41.682359 21146 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:37:41.692901 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 21069
2026-09-27T02:37:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1278 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476410933508-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:37:41 dist-test-slave-grlk krb5kdc[21168](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:37:41 dist-test-slave-grlk krb5kdc[21168](info): set up 2 sockets
Sep 27 02:37:41 dist-test-slave-grlk krb5kdc[21168](info): commencing operation
krb5kdc: starting...
W20260927 02:37:42.636682 392 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476410933508-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476410933508-392-0/krb5kdc/test-user.keytab.
Sep 27 02:37:42 dist-test-slave-grlk krb5kdc[21168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:37:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:37:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476410933508-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476410933508-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:37:42.745136 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44751
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476410933508-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:37:42.849256 21181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:37:42.849483 21181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:37:42.849535 21181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:37:42.853395 21181 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:37:42.853452 21181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:37:42.853478 21181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:37:42.853500 21181 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:37:42.853528 21181 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:37:42.859989 21181 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476410933508-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:37:42.861184 21181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:37:42.862385 21181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:37:42.870306 21189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:42.870301 21187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:42.870311 21186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:42.870569 21181 server_base.cc:1034] running on GCE node
I20260927 02:37:42.871026 21181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:37:42.871960 21181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:37:42.873096 21181 hybrid_clock.cc:648] HybridClock initialized: now 1790476662873074 us; error 34 us; skew 500 ppm
Sep 27 02:37:42 dist-test-slave-grlk krb5kdc[21168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:37:42.875540 21181 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:37:42.876240 21181 webserver.cc:460] Webserver started at http://127.0.0.1:33425/ using document root <none> and password file <none>
I20260927 02:37:42.876688 21181 fs_manager.cc:359] Metadata directory not provided
I20260927 02:37:42.876740 21181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:37:42.876886 21181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:37:42.878677 21181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "a0ec07fc08824e8b900134f5e321660a"
format_stamp: "Formatted at 2026-09-27 02:37:42 on dist-test-slave-grlk"
server_key: "16eed02e54e75d8efb3b6ed176e19d9f"
server_key_iv: "2e4d575308250e258cb69bffdf4def21"
server_key_version: "encryptionkey@0"
I20260927 02:37:42.879120 21181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "a0ec07fc08824e8b900134f5e321660a"
format_stamp: "Formatted at 2026-09-27 02:37:42 on dist-test-slave-grlk"
server_key: "16eed02e54e75d8efb3b6ed176e19d9f"
server_key_iv: "2e4d575308250e258cb69bffdf4def21"
server_key_version: "encryptionkey@0"
I20260927 02:37:42.882391 21181 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:37:42.884444 21196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:42.885533 21181 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:37:42.885622 21181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "a0ec07fc08824e8b900134f5e321660a"
format_stamp: "Formatted at 2026-09-27 02:37:42 on dist-test-slave-grlk"
server_key: "16eed02e54e75d8efb3b6ed176e19d9f"
server_key_iv: "2e4d575308250e258cb69bffdf4def21"
server_key_version: "encryptionkey@0"
I20260927 02:37:42.885696 21181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:37:42.889569 21181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:37:42.892019 21181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:37:42.892140 21181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:37:42.898397 21181 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44751
I20260927 02:37:42.898442 21248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44751 every 8 connection(s)
I20260927 02:37:42.899171 21181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:37:42.899884 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 21181
I20260927 02:37:42.900027 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:37:42.900301 392 external_mini_cluster.cc:1428] Setting key 3cc4fa047ecd77a4d11144fb5ccbb7b5
I20260927 02:37:42.901948 21249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:37:42.904882 21252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:42.901457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45543 (local address 127.0.0.1:44751)
0927 02:37:42.901813 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:42.901830 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:42.902022 (+ 192us) server_negotiation.cc:407] Connection header received
0927 02:37:42.902692 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:42.902707 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:42.902947 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:42.903219 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:42.903878 (+ 659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:42.903883 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:42.903893 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:37:42.903903 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:42.903948 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:37:42.903966 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:42.904078 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:42.904144 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 02:37:42.904277 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":91,"threads_started":1}
I20260927 02:37:42.907312 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:37:42.907495 21249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0ec07fc08824e8b900134f5e321660a: Bootstrap starting.
I20260927 02:37:42.909657 21249 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0ec07fc08824e8b900134f5e321660a: Neither blocks nor log segments found. Creating new log.
I20260927 02:37:42.910492 21249 log.cc:824] T 00000000000000000000000000000000 P a0ec07fc08824e8b900134f5e321660a: Log is configured to *not* fsync() on all Append() calls
I20260927 02:37:42.910600 21252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:42.909071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34137 (local address 127.0.0.1:44751)
0927 02:37:42.909180 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:42.909184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:42.909414 (+ 230us) server_negotiation.cc:407] Connection header received
0927 02:37:42.909518 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:42.909520 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:42.909545 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:42.909604 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:42.910218 (+ 614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:42.910220 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:42.910222 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:37:42.910232 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:42.910254 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:37:42.910264 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:42.910321 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:42.910419 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 02:37:42.910448 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 02:37:42.912314 21249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0ec07fc08824e8b900134f5e321660a: No bootstrap required, opened a new log
I20260927 02:37:42.915798 21249 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0ec07fc08824e8b900134f5e321660a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0ec07fc08824e8b900134f5e321660a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44751 } }
I20260927 02:37:42.916008 21249 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0ec07fc08824e8b900134f5e321660a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:37:42.916064 21249 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0ec07fc08824e8b900134f5e321660a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0ec07fc08824e8b900134f5e321660a, State: Initialized, Role: FOLLOWER
I20260927 02:37:42.916440 21249 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0ec07fc08824e8b900134f5e321660a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0ec07fc08824e8b900134f5e321660a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44751 } }
I20260927 02:37:42.916559 21249 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0ec07fc08824e8b900134f5e321660a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:37:42.916618 21249 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0ec07fc08824e8b900134f5e321660a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:37:42.916708 21249 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0ec07fc08824e8b900134f5e321660a [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:37:42.917580 21249 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0ec07fc08824e8b900134f5e321660a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0ec07fc08824e8b900134f5e321660a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44751 } }
I20260927 02:37:42.917888 21249 leader_election.cc:302] T 00000000000000000000000000000000 P a0ec07fc08824e8b900134f5e321660a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0ec07fc08824e8b900134f5e321660a; no voters:
I20260927 02:37:42.918152 21249 leader_election.cc:288] T 00000000000000000000000000000000 P a0ec07fc08824e8b900134f5e321660a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:37:42.918244 21259 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0ec07fc08824e8b900134f5e321660a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:37:42.918455 21259 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0ec07fc08824e8b900134f5e321660a [term 1 LEADER]: Becoming Leader. State: Replica: a0ec07fc08824e8b900134f5e321660a, State: Running, Role: LEADER
I20260927 02:37:42.918807 21259 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0ec07fc08824e8b900134f5e321660a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0ec07fc08824e8b900134f5e321660a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44751 } }
I20260927 02:37:42.919248 21249 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0ec07fc08824e8b900134f5e321660a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:37:42.920215 21261 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0ec07fc08824e8b900134f5e321660a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0ec07fc08824e8b900134f5e321660a. Latest consensus state: current_term: 1 leader_uuid: "a0ec07fc08824e8b900134f5e321660a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0ec07fc08824e8b900134f5e321660a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44751 } } }
I20260927 02:37:42.920320 21261 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0ec07fc08824e8b900134f5e321660a [sys.catalog]: This master's current role is: LEADER
I20260927 02:37:42.920606 21260 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0ec07fc08824e8b900134f5e321660a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0ec07fc08824e8b900134f5e321660a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0ec07fc08824e8b900134f5e321660a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44751 } } }
I20260927 02:37:42.920708 21260 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0ec07fc08824e8b900134f5e321660a [sys.catalog]: This master's current role is: LEADER
I20260927 02:37:42.920656 21268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:37:42.924065 21268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:37:42.929126 21268 catalog_manager.cc:1347] Generated new cluster ID: 2dd82a579046484a947c6be4719f1b85
I20260927 02:37:42.929178 21268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:37:42.936518 21268 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:37:42.936818 21268 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:37:42.950745 21268 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0ec07fc08824e8b900134f5e321660a: Generated new TSK 0
I20260927 02:37:42.951471 21268 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:37:42.959111 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 21181
2026-09-27T02:37:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1265 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476410933508-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:37:43 dist-test-slave-grlk krb5kdc[21280](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:37:43 dist-test-slave-grlk krb5kdc[21280](info): set up 2 sockets
Sep 27 02:37:43 dist-test-slave-grlk krb5kdc[21280](info): commencing operation
krb5kdc: starting...
W20260927 02:37:43.904656 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476410933508-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476410933508-392-0/krb5kdc/test-user.keytab.
Sep 27 02:37:43 dist-test-slave-grlk krb5kdc[21280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:37:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:37:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476410933508-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476410933508-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:37:44.013806 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41397
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476410933508-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:37:44.116779 21293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:37:44.117022 21293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:37:44.117058 21293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:37:44.120851 21293 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:37:44.120901 21293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:37:44.120918 21293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:37:44.120954 21293 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:37:44.120972 21293 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:37:44.125962 21293 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41397
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476410933508-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:37:44.126801 21293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:37:44.127653 21293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:37:44.135069 21299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:44.135239 21293 server_base.cc:1034] running on GCE node
W20260927 02:37:44.135124 21301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:44.135178 21298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:44.135830 21293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:37:44.136775 21293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:37:44.137923 21293 hybrid_clock.cc:648] HybridClock initialized: now 1790476664137896 us; error 37 us; skew 500 ppm
Sep 27 02:37:44 dist-test-slave-grlk krb5kdc[21280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476664, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:37:44.140375 21293 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:37:44.141160 21293 webserver.cc:460] Webserver started at http://127.0.0.1:32981/ using document root <none> and password file <none>
I20260927 02:37:44.141635 21293 fs_manager.cc:359] Metadata directory not provided
I20260927 02:37:44.141690 21293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:37:44.141847 21293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:37:44.143654 21293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "5f422a43e2b94339ae24d822ace22b6e"
format_stamp: "Formatted at 2026-09-27 02:37:44 on dist-test-slave-grlk"
server_key: "8d18f744bfdadd9aaf1f512c99e1baf5"
server_key_iv: "f98e77205ff27f6e96762cdc1ef596dc"
server_key_version: "encryptionkey@0"
I20260927 02:37:44.144126 21293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "5f422a43e2b94339ae24d822ace22b6e"
format_stamp: "Formatted at 2026-09-27 02:37:44 on dist-test-slave-grlk"
server_key: "8d18f744bfdadd9aaf1f512c99e1baf5"
server_key_iv: "f98e77205ff27f6e96762cdc1ef596dc"
server_key_version: "encryptionkey@0"
I20260927 02:37:44.147325 21293 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 02:37:44.149462 21308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:44.150496 21293 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:37:44.150588 21293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "5f422a43e2b94339ae24d822ace22b6e"
format_stamp: "Formatted at 2026-09-27 02:37:44 on dist-test-slave-grlk"
server_key: "8d18f744bfdadd9aaf1f512c99e1baf5"
server_key_iv: "f98e77205ff27f6e96762cdc1ef596dc"
server_key_version: "encryptionkey@0"
I20260927 02:37:44.150674 21293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:37:44.165856 21293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:37:44.169248 21293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:37:44.169418 21293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:37:44.175562 21293 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41397
I20260927 02:37:44.175572 21360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41397 every 8 connection(s)
I20260927 02:37:44.176405 21293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:37:44.178375 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 21293
I20260927 02:37:44.178503 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:37:44.178735 392 external_mini_cluster.cc:1428] Setting key a732dd6e95f0f7b085357b06b3cb90df
I20260927 02:37:44.179009 21361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:37:44.185482 21361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f422a43e2b94339ae24d822ace22b6e: Bootstrap starting.
Sep 27 02:37:44 dist-test-slave-grlk krb5kdc[21280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 02:37:44.187503 21361 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f422a43e2b94339ae24d822ace22b6e: Neither blocks nor log segments found. Creating new log.
I20260927 02:37:44.188285 21361 log.cc:824] T 00000000000000000000000000000000 P 5f422a43e2b94339ae24d822ace22b6e: Log is configured to *not* fsync() on all Append() calls
I20260927 02:37:44.190254 21361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f422a43e2b94339ae24d822ace22b6e: No bootstrap required, opened a new log
I20260927 02:37:44.192209 21364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:44.179957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:56177 (local address 127.0.0.1:41397)
0927 02:37:44.180342 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:44.180358 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:44.180412 (+ 54us) server_negotiation.cc:407] Connection header received
0927 02:37:44.181117 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:44.181138 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:44.181552 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:44.181888 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:44.182467 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:44.183874 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:44.184644 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:44.185330 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:44.185487 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:44.187764 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:44.187770 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:44.187784 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:37:44.187812 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:44.189451 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:44.189881 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:44.189887 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:44.189892 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:44.190018 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:44.190282 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:44.190285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:44.190286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:44.190735 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:44.190919 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:44.191157 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 02:37:44.191357 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":104,"threads_started":1}
I20260927 02:37:44.194212 21361 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f422a43e2b94339ae24d822ace22b6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f422a43e2b94339ae24d822ace22b6e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41397 } }
I20260927 02:37:44.194458 21361 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f422a43e2b94339ae24d822ace22b6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:37:44.194518 21361 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f422a43e2b94339ae24d822ace22b6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f422a43e2b94339ae24d822ace22b6e, State: Initialized, Role: FOLLOWER
I20260927 02:37:44.194891 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:37:44.194918 21361 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f422a43e2b94339ae24d822ace22b6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f422a43e2b94339ae24d822ace22b6e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41397 } }
I20260927 02:37:44.195056 21361 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f422a43e2b94339ae24d822ace22b6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:37:44.195114 21361 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f422a43e2b94339ae24d822ace22b6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:37:44.195201 21361 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f422a43e2b94339ae24d822ace22b6e [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:37:44.195961 21361 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f422a43e2b94339ae24d822ace22b6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f422a43e2b94339ae24d822ace22b6e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41397 } }
I20260927 02:37:44.196261 21361 leader_election.cc:302] T 00000000000000000000000000000000 P 5f422a43e2b94339ae24d822ace22b6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f422a43e2b94339ae24d822ace22b6e; no voters:
I20260927 02:37:44.196517 21361 leader_election.cc:288] T 00000000000000000000000000000000 P 5f422a43e2b94339ae24d822ace22b6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:37:44.196616 21370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f422a43e2b94339ae24d822ace22b6e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:37:44.196959 21370 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f422a43e2b94339ae24d822ace22b6e [term 1 LEADER]: Becoming Leader. State: Replica: 5f422a43e2b94339ae24d822ace22b6e, State: Running, Role: LEADER
I20260927 02:37:44.197288 21370 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f422a43e2b94339ae24d822ace22b6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f422a43e2b94339ae24d822ace22b6e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41397 } }
I20260927 02:37:44.197752 21361 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f422a43e2b94339ae24d822ace22b6e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:37:44.198688 21372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f422a43e2b94339ae24d822ace22b6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f422a43e2b94339ae24d822ace22b6e. Latest consensus state: current_term: 1 leader_uuid: "5f422a43e2b94339ae24d822ace22b6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f422a43e2b94339ae24d822ace22b6e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41397 } } }
I20260927 02:37:44.198804 21372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f422a43e2b94339ae24d822ace22b6e [sys.catalog]: This master's current role is: LEADER
I20260927 02:37:44.199177 21380 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:37:44.199159 21373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f422a43e2b94339ae24d822ace22b6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f422a43e2b94339ae24d822ace22b6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f422a43e2b94339ae24d822ace22b6e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41397 } } }
I20260927 02:37:44.199889 21373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f422a43e2b94339ae24d822ace22b6e [sys.catalog]: This master's current role is: LEADER
I20260927 02:37:44.202294 21380 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:37:44.208069 21380 catalog_manager.cc:1347] Generated new cluster ID: 1243cc1594dd478ba7ddcb54f7844e5d
I20260927 02:37:44.208148 21380 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:37:44.208249 21364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:44.196662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:55819 (local address 127.0.0.1:41397)
0927 02:37:44.196760 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:44.196764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:44.199813 (+ 3049us) server_negotiation.cc:407] Connection header received
0927 02:37:44.199951 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:44.199954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:44.199999 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:44.200081 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:44.200717 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:44.201500 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:44.202596 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:44.202938 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:44.203012 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:44.204860 (+ 1848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:44.204863 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:44.204866 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:37:44.204890 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:44.206372 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:44.206875 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:44.206887 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:44.206889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:44.206975 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:44.207312 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:44.207315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:44.207317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:44.207512 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:44.207648 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:44.207933 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 02:37:44.208026 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 02:37:44.219373 21380 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:37:44.220515 21380 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:37:44.227926 21380 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f422a43e2b94339ae24d822ace22b6e: Generated new TSK 0
I20260927 02:37:44.228582 21380 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:37:44.231940 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 21293
W20260927 02:37:44.236515 21278 connection.cc:535] client connection to 127.0.0.1:41397 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41397): Connection reset by peer (error 104)
2026-09-27T02:37:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1273 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476410933508-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:37:44 dist-test-slave-grlk krb5kdc[21392](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:37:44 dist-test-slave-grlk krb5kdc[21392](info): set up 2 sockets
Sep 27 02:37:44 dist-test-slave-grlk krb5kdc[21392](info): commencing operation
krb5kdc: starting...
W20260927 02:37:45.180709 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476410933508-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476410933508-392-0/krb5kdc/test-user.keytab.
Sep 27 02:37:45 dist-test-slave-grlk krb5kdc[21392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:37:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:37:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476410933508-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476410933508-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:37:45.291054 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33823
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476410933508-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33823
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:37:45.394312 21405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:37:45.394572 21405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:37:45.394618 21405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:37:45.398537 21405 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:37:45.398589 21405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:37:45.398607 21405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:37:45.398622 21405 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:37:45.398638 21405 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:37:45.403739 21405 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476410933508-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:37:45.404646 21405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:37:45.405610 21405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:37:45.413447 21411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:45.413647 21405 server_base.cc:1034] running on GCE node
W20260927 02:37:45.413442 21410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:45.413487 21413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:45.414278 21405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:37:45.415264 21405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:37:45.416414 21405 hybrid_clock.cc:648] HybridClock initialized: now 1790476665416395 us; error 32 us; skew 500 ppm
Sep 27 02:37:45 dist-test-slave-grlk krb5kdc[21392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476665, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:37:45.418848 21405 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:37:45.419672 21405 webserver.cc:460] Webserver started at http://127.0.0.1:33883/ using document root <none> and password file <none>
I20260927 02:37:45.420166 21405 fs_manager.cc:359] Metadata directory not provided
I20260927 02:37:45.420228 21405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:37:45.420393 21405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:37:45.422279 21405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "8c676f3ea8224c488f9b23c0910ef785"
format_stamp: "Formatted at 2026-09-27 02:37:45 on dist-test-slave-grlk"
server_key: "e4812687a25c2b175d655a7c55c99280"
server_key_iv: "40d9a15fc20291660f3d65aa576ece5d"
server_key_version: "encryptionkey@0"
I20260927 02:37:45.422773 21405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "8c676f3ea8224c488f9b23c0910ef785"
format_stamp: "Formatted at 2026-09-27 02:37:45 on dist-test-slave-grlk"
server_key: "e4812687a25c2b175d655a7c55c99280"
server_key_iv: "40d9a15fc20291660f3d65aa576ece5d"
server_key_version: "encryptionkey@0"
I20260927 02:37:45.426155 21405 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:37:45.428318 21420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:45.429455 21405 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:37:45.429561 21405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "8c676f3ea8224c488f9b23c0910ef785"
format_stamp: "Formatted at 2026-09-27 02:37:45 on dist-test-slave-grlk"
server_key: "e4812687a25c2b175d655a7c55c99280"
server_key_iv: "40d9a15fc20291660f3d65aa576ece5d"
server_key_version: "encryptionkey@0"
I20260927 02:37:45.429651 21405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:37:45.460035 21405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:37:45.463021 21405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:37:45.463189 21405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:37:45.469729 21405 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33823
I20260927 02:37:45.469777 21472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33823 every 8 connection(s)
I20260927 02:37:45.470621 21405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:37:45.473269 21473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:37:45.476332 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 21405
I20260927 02:37:45.476485 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:37:45.476783 392 external_mini_cluster.cc:1428] Setting key ceab0cad8876013d774f70567fe3b8aa
I20260927 02:37:45.479211 21473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c676f3ea8224c488f9b23c0910ef785: Bootstrap starting.
I20260927 02:37:45.481375 21473 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c676f3ea8224c488f9b23c0910ef785: Neither blocks nor log segments found. Creating new log.
I20260927 02:37:45.482076 21473 log.cc:824] T 00000000000000000000000000000000 P 8c676f3ea8224c488f9b23c0910ef785: Log is configured to *not* fsync() on all Append() calls
I20260927 02:37:45.483805 21473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c676f3ea8224c488f9b23c0910ef785: No bootstrap required, opened a new log
I20260927 02:37:45.485090 21476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:45.478192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:54869 (local address 127.0.0.1:33823)
0927 02:37:45.478605 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:45.478619 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:45.478653 (+ 34us) server_negotiation.cc:407] Connection header received
0927 02:37:45.479268 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:45.479289 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:45.479631 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:45.479929 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:45.480310 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:45.481289 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:45.482156 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:45.482625 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:45.482771 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:45.483516 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:45.483521 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:45.483533 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:37:45.483914 (+ 381us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:45.483978 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:37:45.484002 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:45.484111 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:45.484282 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 02:37:45.484402 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":135,"threads_started":1}
I20260927 02:37:45.487295 21473 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c676f3ea8224c488f9b23c0910ef785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c676f3ea8224c488f9b23c0910ef785" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33823 } }
I20260927 02:37:45.487509 21473 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c676f3ea8224c488f9b23c0910ef785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:37:45.487612 21473 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c676f3ea8224c488f9b23c0910ef785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c676f3ea8224c488f9b23c0910ef785, State: Initialized, Role: FOLLOWER
I20260927 02:37:45.487717 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:37:45.488031 21473 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c676f3ea8224c488f9b23c0910ef785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c676f3ea8224c488f9b23c0910ef785" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33823 } }
I20260927 02:37:45.488165 21473 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c676f3ea8224c488f9b23c0910ef785 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:37:45.488234 21473 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c676f3ea8224c488f9b23c0910ef785 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:37:45.488332 21473 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c676f3ea8224c488f9b23c0910ef785 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:37:45.489307 21473 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c676f3ea8224c488f9b23c0910ef785 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c676f3ea8224c488f9b23c0910ef785" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33823 } }
I20260927 02:37:45.489691 21473 leader_election.cc:302] T 00000000000000000000000000000000 P 8c676f3ea8224c488f9b23c0910ef785 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c676f3ea8224c488f9b23c0910ef785; no voters:
I20260927 02:37:45.490002 21473 leader_election.cc:288] T 00000000000000000000000000000000 P 8c676f3ea8224c488f9b23c0910ef785 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:37:45.490159 21483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c676f3ea8224c488f9b23c0910ef785 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:37:45.490343 21483 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c676f3ea8224c488f9b23c0910ef785 [term 1 LEADER]: Becoming Leader. State: Replica: 8c676f3ea8224c488f9b23c0910ef785, State: Running, Role: LEADER
I20260927 02:37:45.490623 21483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c676f3ea8224c488f9b23c0910ef785 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c676f3ea8224c488f9b23c0910ef785" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33823 } }
I20260927 02:37:45.491175 21473 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c676f3ea8224c488f9b23c0910ef785 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:37:45.492159 21484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c676f3ea8224c488f9b23c0910ef785 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c676f3ea8224c488f9b23c0910ef785" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c676f3ea8224c488f9b23c0910ef785" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33823 } } }
I20260927 02:37:45.492153 21485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c676f3ea8224c488f9b23c0910ef785 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c676f3ea8224c488f9b23c0910ef785. Latest consensus state: current_term: 1 leader_uuid: "8c676f3ea8224c488f9b23c0910ef785" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c676f3ea8224c488f9b23c0910ef785" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33823 } } }
I20260927 02:37:45.492283 21485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c676f3ea8224c488f9b23c0910ef785 [sys.catalog]: This master's current role is: LEADER
I20260927 02:37:45.492283 21484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c676f3ea8224c488f9b23c0910ef785 [sys.catalog]: This master's current role is: LEADER
I20260927 02:37:45.493431 21492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:37:45.496212 21476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:45.489594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:35029 (local address 127.0.0.1:33823)
0927 02:37:45.489691 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:45.489694 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:45.492448 (+ 2754us) server_negotiation.cc:407] Connection header received
0927 02:37:45.492604 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:45.492607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:45.492656 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:45.492728 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:45.493151 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:45.493869 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:45.494742 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:45.494972 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:45.495054 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:45.495789 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:45.495792 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:45.495795 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:37:45.495807 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:45.495832 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:37:45.495842 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:45.495905 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:45.496055 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 02:37:45.496087 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 02:37:45.496320 21492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:37:45.501740 21492 catalog_manager.cc:1347] Generated new cluster ID: 3a75ac15e7bb477ea93ec40a126270a7
I20260927 02:37:45.501804 21492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:37:45.517599 21492 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:37:45.518528 21492 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:37:45.524185 21492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c676f3ea8224c488f9b23c0910ef785: Generated new TSK 0
I20260927 02:37:45.524923 21492 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:37:45.534377 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 21405
W20260927 02:37:45.538869 21390 connection.cc:535] client connection to 127.0.0.1:33823 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33823): Connection reset by peer (error 104)
2026-09-27T02:37:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1302 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476410933508-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:37:45 dist-test-slave-grlk krb5kdc[21504](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:37:45 dist-test-slave-grlk krb5kdc[21504](info): set up 2 sockets
Sep 27 02:37:45 dist-test-slave-grlk krb5kdc[21504](info): commencing operation
krb5kdc: starting...
W20260927 02:37:46.485015 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476410933508-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476410933508-392-0/krb5kdc/test-user.keytab.
Sep 27 02:37:46 dist-test-slave-grlk krb5kdc[21504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:37:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:37:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476410933508-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476410933508-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:37:46.612800 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45263
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476410933508-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45263
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:37:46.731909 21517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:37:46.732254 21517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:37:46.732465 21517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:37:46.737323 21517 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:37:46.737417 21517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:37:46.737447 21517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:37:46.737468 21517 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:37:46.737489 21517 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:37:46.743073 21517 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476410933508-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:37:46.744113 21517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:37:46.745160 21517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:37:46.753474 21517 server_base.cc:1034] running on GCE node
W20260927 02:37:46.753511 21523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:46.753615 21525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:46.753515 21522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:46.754248 21517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:37:46.755637 21517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:37:46.756858 21517 hybrid_clock.cc:648] HybridClock initialized: now 1790476666756832 us; error 36 us; skew 500 ppm
Sep 27 02:37:46 dist-test-slave-grlk krb5kdc[21504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:37:46.764082 21517 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:37:46.765043 21517 webserver.cc:460] Webserver started at http://127.0.0.1:33093/ using document root <none> and password file <none>
I20260927 02:37:46.765580 21517 fs_manager.cc:359] Metadata directory not provided
I20260927 02:37:46.765642 21517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:37:46.765820 21517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:37:46.767774 21517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "3d8946a59c764e89ab84347db68d071b"
format_stamp: "Formatted at 2026-09-27 02:37:46 on dist-test-slave-grlk"
server_key: "f3220351f0651fe4558a8a1ef2268f6f"
server_key_iv: "3f6a974049dd5913f52f9a093c542f35"
server_key_version: "encryptionkey@0"
I20260927 02:37:46.768266 21517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "3d8946a59c764e89ab84347db68d071b"
format_stamp: "Formatted at 2026-09-27 02:37:46 on dist-test-slave-grlk"
server_key: "f3220351f0651fe4558a8a1ef2268f6f"
server_key_iv: "3f6a974049dd5913f52f9a093c542f35"
server_key_version: "encryptionkey@0"
I20260927 02:37:46.771755 21517 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260927 02:37:46.774036 21532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:46.775645 21517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 02:37:46.775808 21517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "3d8946a59c764e89ab84347db68d071b"
format_stamp: "Formatted at 2026-09-27 02:37:46 on dist-test-slave-grlk"
server_key: "f3220351f0651fe4558a8a1ef2268f6f"
server_key_iv: "3f6a974049dd5913f52f9a093c542f35"
server_key_version: "encryptionkey@0"
I20260927 02:37:46.775918 21517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:37:46.780434 21517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:37:46.783504 21517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:37:46.783665 21517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:37:46.790198 21517 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45263
I20260927 02:37:46.790210 21584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45263 every 8 connection(s)
I20260927 02:37:46.791052 21517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:37:46.793594 21585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:37:46.798671 21585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d8946a59c764e89ab84347db68d071b: Bootstrap starting.
I20260927 02:37:46.799010 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 21517
I20260927 02:37:46.799135 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:37:46.799379 392 external_mini_cluster.cc:1428] Setting key d908297bda4f35ce7fa0a034d80ca545
I20260927 02:37:46.801067 21585 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d8946a59c764e89ab84347db68d071b: Neither blocks nor log segments found. Creating new log.
I20260927 02:37:46.801733 21585 log.cc:824] T 00000000000000000000000000000000 P 3d8946a59c764e89ab84347db68d071b: Log is configured to *not* fsync() on all Append() calls
I20260927 02:37:46.803826 21585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d8946a59c764e89ab84347db68d071b: No bootstrap required, opened a new log
I20260927 02:37:46.804366 21588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:46.800553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:41551 (local address 127.0.0.1:45263)
0927 02:37:46.801000 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:46.801016 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:46.801049 (+ 33us) server_negotiation.cc:407] Connection header received
0927 02:37:46.801663 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:46.801678 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:46.801966 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:46.802318 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:46.802976 (+ 658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:46.802981 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:46.802996 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:37:46.803006 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:46.803067 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:37:46.803094 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:46.803203 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:46.803313 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 02:37:46.803466 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":120,"threads_started":1}
I20260927 02:37:46.807118 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:37:46.807806 21585 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d8946a59c764e89ab84347db68d071b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8946a59c764e89ab84347db68d071b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45263 } }
I20260927 02:37:46.808012 21585 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d8946a59c764e89ab84347db68d071b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:37:46.808055 21585 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d8946a59c764e89ab84347db68d071b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d8946a59c764e89ab84347db68d071b, State: Initialized, Role: FOLLOWER
I20260927 02:37:46.808470 21585 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d8946a59c764e89ab84347db68d071b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8946a59c764e89ab84347db68d071b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45263 } }
I20260927 02:37:46.808593 21585 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d8946a59c764e89ab84347db68d071b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:37:46.808650 21585 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d8946a59c764e89ab84347db68d071b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:37:46.808736 21585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d8946a59c764e89ab84347db68d071b [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:37:46.809808 21585 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d8946a59c764e89ab84347db68d071b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8946a59c764e89ab84347db68d071b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45263 } }
I20260927 02:37:46.810154 21585 leader_election.cc:302] T 00000000000000000000000000000000 P 3d8946a59c764e89ab84347db68d071b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d8946a59c764e89ab84347db68d071b; no voters:
I20260927 02:37:46.810469 21585 leader_election.cc:288] T 00000000000000000000000000000000 P 3d8946a59c764e89ab84347db68d071b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:37:46.810506 21588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:46.809036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:41855 (local address 127.0.0.1:45263)
0927 02:37:46.809144 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:46.809148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:46.809304 (+ 156us) server_negotiation.cc:407] Connection header received
0927 02:37:46.809390 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:46.809393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:46.809428 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:46.809498 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:46.810152 (+ 654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:46.810155 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:46.810157 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:37:46.810167 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:46.810191 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:37:46.810201 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:46.810268 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:46.810353 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 02:37:46.810385 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 02:37:46.810563 21595 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d8946a59c764e89ab84347db68d071b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:37:46.810753 21595 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d8946a59c764e89ab84347db68d071b [term 1 LEADER]: Becoming Leader. State: Replica: 3d8946a59c764e89ab84347db68d071b, State: Running, Role: LEADER
I20260927 02:37:46.811045 21595 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d8946a59c764e89ab84347db68d071b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8946a59c764e89ab84347db68d071b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45263 } }
I20260927 02:37:46.811738 21585 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d8946a59c764e89ab84347db68d071b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:37:46.812189 21595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d8946a59c764e89ab84347db68d071b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d8946a59c764e89ab84347db68d071b. Latest consensus state: current_term: 1 leader_uuid: "3d8946a59c764e89ab84347db68d071b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8946a59c764e89ab84347db68d071b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45263 } } }
I20260927 02:37:46.812300 21595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d8946a59c764e89ab84347db68d071b [sys.catalog]: This master's current role is: LEADER
I20260927 02:37:46.812531 21595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d8946a59c764e89ab84347db68d071b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d8946a59c764e89ab84347db68d071b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8946a59c764e89ab84347db68d071b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45263 } } }
I20260927 02:37:46.812676 21595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d8946a59c764e89ab84347db68d071b [sys.catalog]: This master's current role is: LEADER
I20260927 02:37:46.812686 21600 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:37:46.816530 21600 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:37:46.821797 21600 catalog_manager.cc:1347] Generated new cluster ID: 8212ac8ccb1f46b18d414c93fe82ba15
I20260927 02:37:46.821918 21600 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:37:46.839118 21600 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:37:46.839497 21600 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:37:46.846241 21600 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d8946a59c764e89ab84347db68d071b: Generated new TSK 0
I20260927 02:37:46.846984 21600 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:37:46.854666 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 21517
2026-09-27T02:37:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1320 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476410933508-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:37:46 dist-test-slave-grlk krb5kdc[21616](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:37:46 dist-test-slave-grlk krb5kdc[21616](info): set up 2 sockets
Sep 27 02:37:46 dist-test-slave-grlk krb5kdc[21616](info): commencing operation
krb5kdc: starting...
W20260927 02:37:47.817175 392 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476410933508-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476410933508-392-0/krb5kdc/test-user.keytab.
Sep 27 02:37:47 dist-test-slave-grlk krb5kdc[21616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:37:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:37:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476410933508-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476410933508-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:37:47.929209 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35757
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476410933508-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35757
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:37:48.034554 21629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:37:48.034813 21629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:37:48.034870 21629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:37:48.038831 21629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:37:48.038899 21629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:37:48.038926 21629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:37:48.038947 21629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:37:48.038976 21629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:37:48.044162 21629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35757
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476410933508-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:37:48.045189 21629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:37:48.046145 21629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:37:48.054033 21637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:48.054013 21634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:48.054070 21629 server_base.cc:1034] running on GCE node
W20260927 02:37:48.054013 21635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:48.054730 21629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:37:48.055697 21629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:37:48.056847 21629 hybrid_clock.cc:648] HybridClock initialized: now 1790476668056831 us; error 27 us; skew 500 ppm
Sep 27 02:37:48 dist-test-slave-grlk krb5kdc[21616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:37:48.059247 21629 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:37:48.059986 21629 webserver.cc:460] Webserver started at http://127.0.0.1:43307/ using document root <none> and password file <none>
I20260927 02:37:48.060458 21629 fs_manager.cc:359] Metadata directory not provided
I20260927 02:37:48.060509 21629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:37:48.060662 21629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:37:48.062544 21629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "b66a1bbc05e84631813a72bba4bb2ba3"
format_stamp: "Formatted at 2026-09-27 02:37:48 on dist-test-slave-grlk"
server_key: "79dca63e2643ed3a13719697f5398140"
server_key_iv: "3ee8544ebe522b01d957df7c5d7a6c2a"
server_key_version: "encryptionkey@0"
I20260927 02:37:48.062999 21629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "b66a1bbc05e84631813a72bba4bb2ba3"
format_stamp: "Formatted at 2026-09-27 02:37:48 on dist-test-slave-grlk"
server_key: "79dca63e2643ed3a13719697f5398140"
server_key_iv: "3ee8544ebe522b01d957df7c5d7a6c2a"
server_key_version: "encryptionkey@0"
I20260927 02:37:48.066344 21629 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260927 02:37:48.068394 21644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:48.069514 21629 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260927 02:37:48.069607 21629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "b66a1bbc05e84631813a72bba4bb2ba3"
format_stamp: "Formatted at 2026-09-27 02:37:48 on dist-test-slave-grlk"
server_key: "79dca63e2643ed3a13719697f5398140"
server_key_iv: "3ee8544ebe522b01d957df7c5d7a6c2a"
server_key_version: "encryptionkey@0"
I20260927 02:37:48.069685 21629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:37:48.086625 21629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:37:48.089083 21629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:37:48.089214 21629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:37:48.095391 21629 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35757
I20260927 02:37:48.095418 21696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35757 every 8 connection(s)
I20260927 02:37:48.096210 21629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:37:48.098724 21697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:37:48.103855 21697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b66a1bbc05e84631813a72bba4bb2ba3: Bootstrap starting.
I20260927 02:37:48.104223 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 21629
I20260927 02:37:48.104344 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:37:48.104602 392 external_mini_cluster.cc:1428] Setting key 53f68c140c69c710395bbcbddf13ab6a
I20260927 02:37:48.106293 21697 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b66a1bbc05e84631813a72bba4bb2ba3: Neither blocks nor log segments found. Creating new log.
I20260927 02:37:48.106976 21697 log.cc:824] T 00000000000000000000000000000000 P b66a1bbc05e84631813a72bba4bb2ba3: Log is configured to *not* fsync() on all Append() calls
I20260927 02:37:48.108865 21697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b66a1bbc05e84631813a72bba4bb2ba3: No bootstrap required, opened a new log
I20260927 02:37:48.113058 21697 raft_consensus.cc:348] T 00000000000000000000000000000000 P b66a1bbc05e84631813a72bba4bb2ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66a1bbc05e84631813a72bba4bb2ba3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35757 } }
I20260927 02:37:48.113276 21697 raft_consensus.cc:374] T 00000000000000000000000000000000 P b66a1bbc05e84631813a72bba4bb2ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:37:48.113318 21697 raft_consensus.cc:729] T 00000000000000000000000000000000 P b66a1bbc05e84631813a72bba4bb2ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b66a1bbc05e84631813a72bba4bb2ba3, State: Initialized, Role: FOLLOWER
Sep 27 02:37:48 dist-test-slave-grlk krb5kdc[21616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 02:37:48.113735 21697 consensus_queue.cc:260] T 00000000000000000000000000000000 P b66a1bbc05e84631813a72bba4bb2ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66a1bbc05e84631813a72bba4bb2ba3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35757 } }
I20260927 02:37:48.113867 21697 raft_consensus.cc:388] T 00000000000000000000000000000000 P b66a1bbc05e84631813a72bba4bb2ba3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:37:48.113936 21697 raft_consensus.cc:482] T 00000000000000000000000000000000 P b66a1bbc05e84631813a72bba4bb2ba3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:37:48.114034 21697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b66a1bbc05e84631813a72bba4bb2ba3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:37:48.114874 21697 raft_consensus.cc:504] T 00000000000000000000000000000000 P b66a1bbc05e84631813a72bba4bb2ba3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66a1bbc05e84631813a72bba4bb2ba3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35757 } }
I20260927 02:37:48.115185 21697 leader_election.cc:302] T 00000000000000000000000000000000 P b66a1bbc05e84631813a72bba4bb2ba3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b66a1bbc05e84631813a72bba4bb2ba3; no voters:
I20260927 02:37:48.115484 21697 leader_election.cc:288] T 00000000000000000000000000000000 P b66a1bbc05e84631813a72bba4bb2ba3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:37:48.115545 21702 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b66a1bbc05e84631813a72bba4bb2ba3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:37:48.115756 21702 raft_consensus.cc:686] T 00000000000000000000000000000000 P b66a1bbc05e84631813a72bba4bb2ba3 [term 1 LEADER]: Becoming Leader. State: Replica: b66a1bbc05e84631813a72bba4bb2ba3, State: Running, Role: LEADER
I20260927 02:37:48.116104 21702 consensus_queue.cc:237] T 00000000000000000000000000000000 P b66a1bbc05e84631813a72bba4bb2ba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66a1bbc05e84631813a72bba4bb2ba3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35757 } }
I20260927 02:37:48.116734 21697 sys_catalog.cc:564] T 00000000000000000000000000000000 P b66a1bbc05e84631813a72bba4bb2ba3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:37:48.117641 21703 sys_catalog.cc:455] T 00000000000000000000000000000000 P b66a1bbc05e84631813a72bba4bb2ba3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b66a1bbc05e84631813a72bba4bb2ba3. Latest consensus state: current_term: 1 leader_uuid: "b66a1bbc05e84631813a72bba4bb2ba3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66a1bbc05e84631813a72bba4bb2ba3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35757 } } }
I20260927 02:37:48.117766 21703 sys_catalog.cc:458] T 00000000000000000000000000000000 P b66a1bbc05e84631813a72bba4bb2ba3 [sys.catalog]: This master's current role is: LEADER
I20260927 02:37:48.117925 21704 sys_catalog.cc:455] T 00000000000000000000000000000000 P b66a1bbc05e84631813a72bba4bb2ba3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b66a1bbc05e84631813a72bba4bb2ba3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66a1bbc05e84631813a72bba4bb2ba3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35757 } } }
I20260927 02:37:48.118224 21704 sys_catalog.cc:458] T 00000000000000000000000000000000 P b66a1bbc05e84631813a72bba4bb2ba3 [sys.catalog]: This master's current role is: LEADER
I20260927 02:37:48.118343 21711 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:37:48.118614 21700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:48.105793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:54561 (local address 127.0.0.1:35757)
0927 02:37:48.106180 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:48.106194 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:48.106225 (+ 31us) server_negotiation.cc:407] Connection header received
0927 02:37:48.106824 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:48.106840 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:48.107165 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:48.107460 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:48.108325 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:48.109313 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:48.110516 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:48.110991 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:48.111118 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:48.113939 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:48.113945 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:48.113960 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:37:48.113989 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:48.115571 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:48.116097 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:48.116103 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:48.116108 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:48.116214 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:48.116532 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:48.116535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:48.116537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:48.116912 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:48.117133 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:48.117404 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 02:37:48.117635 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":114,"threads_started":1}
I20260927 02:37:48.121423 21711 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:37:48.121825 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:37:48.126866 21711 catalog_manager.cc:1347] Generated new cluster ID: 24d79b8592d545d88de94797ec4f3c56
I20260927 02:37:48.126919 21711 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:37:48.132143 21700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:48.123932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:60011 (local address 127.0.0.1:35757)
0927 02:37:48.124033 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:48.124038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:48.124301 (+ 263us) server_negotiation.cc:407] Connection header received
0927 02:37:48.124414 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:48.124417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:48.124460 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:48.124518 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:48.125218 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:48.126352 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:48.127365 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:48.127605 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:48.127692 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:48.129065 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:48.129070 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:48.129072 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:37:48.129098 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:48.130451 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:48.130923 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:48.130927 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:48.130928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:48.131002 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:48.131319 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:48.131322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:48.131324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:48.131488 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:48.131622 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:48.131897 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 02:37:48.131986 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 02:37:48.166996 21711 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:37:48.167860 21711 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:37:48.172046 21711 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b66a1bbc05e84631813a72bba4bb2ba3: Generated new TSK 0
I20260927 02:37:48.172699 21711 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:37:48.176546 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 21629
W20260927 02:37:48.181686 21614 connection.cc:535] client connection to 127.0.0.1:35757 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35757): Connection reset by peer (error 104)
2026-09-27T02:37:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1322 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476410933508-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:37:48 dist-test-slave-grlk krb5kdc[21728](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:37:48 dist-test-slave-grlk krb5kdc[21728](info): set up 2 sockets
Sep 27 02:37:48 dist-test-slave-grlk krb5kdc[21728](info): commencing operation
krb5kdc: starting...
W20260927 02:37:49.136847 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476410933508-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476410933508-392-0/krb5kdc/test-user.keytab.
Sep 27 02:37:49 dist-test-slave-grlk krb5kdc[21728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:37:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:37:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476410933508-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476410933508-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:37:49.245947 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45777
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476410933508-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:37:49.348639 21741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:37:49.348896 21741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:37:49.348981 21741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:37:49.352849 21741 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:37:49.352903 21741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:37:49.352949 21741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:37:49.352974 21741 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:37:49.353003 21741 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:37:49.358083 21741 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45777
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476410933508-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:37:49.359038 21741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:37:49.359968 21741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:37:49.367936 21747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:49.367905 21746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:49.368031 21741 server_base.cc:1034] running on GCE node
W20260927 02:37:49.367955 21749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:49.368621 21741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:37:49.369702 21741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:37:49.370860 21741 hybrid_clock.cc:648] HybridClock initialized: now 1790476669370847 us; error 24 us; skew 500 ppm
Sep 27 02:37:49 dist-test-slave-grlk krb5kdc[21728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476669, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:37:49.373394 21741 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:37:49.374207 21741 webserver.cc:460] Webserver started at http://127.0.0.1:33993/ using document root <none> and password file <none>
I20260927 02:37:49.374701 21741 fs_manager.cc:359] Metadata directory not provided
I20260927 02:37:49.374754 21741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:37:49.374909 21741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:37:49.377103 21741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "83b9d959cc564e969663badea5d58f43"
format_stamp: "Formatted at 2026-09-27 02:37:49 on dist-test-slave-grlk"
server_key: "1dfa769fcffea05f35af0b7dbf92549a"
server_key_iv: "c307dbf671136482e34f7bf30ffb2d05"
server_key_version: "encryptionkey@0"
I20260927 02:37:49.377588 21741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "83b9d959cc564e969663badea5d58f43"
format_stamp: "Formatted at 2026-09-27 02:37:49 on dist-test-slave-grlk"
server_key: "1dfa769fcffea05f35af0b7dbf92549a"
server_key_iv: "c307dbf671136482e34f7bf30ffb2d05"
server_key_version: "encryptionkey@0"
I20260927 02:37:49.380997 21741 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:37:49.383129 21756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:49.384265 21741 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:37:49.384361 21741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "83b9d959cc564e969663badea5d58f43"
format_stamp: "Formatted at 2026-09-27 02:37:49 on dist-test-slave-grlk"
server_key: "1dfa769fcffea05f35af0b7dbf92549a"
server_key_iv: "c307dbf671136482e34f7bf30ffb2d05"
server_key_version: "encryptionkey@0"
I20260927 02:37:49.384461 21741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:37:49.395998 21741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:37:49.398591 21741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:37:49.398743 21741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:37:49.405314 21741 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45777
I20260927 02:37:49.405372 21808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45777 every 8 connection(s)
I20260927 02:37:49.406226 21741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:37:49.408921 21809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:37:49.410530 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 21741
I20260927 02:37:49.410636 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:37:49.410863 392 external_mini_cluster.cc:1428] Setting key 37d05cb5e5d48a751f85215795b87eb0
I20260927 02:37:49.414693 21809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83b9d959cc564e969663badea5d58f43: Bootstrap starting.
I20260927 02:37:49.416765 21809 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83b9d959cc564e969663badea5d58f43: Neither blocks nor log segments found. Creating new log.
I20260927 02:37:49.417587 21809 log.cc:824] T 00000000000000000000000000000000 P 83b9d959cc564e969663badea5d58f43: Log is configured to *not* fsync() on all Append() calls
I20260927 02:37:49.419518 21809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83b9d959cc564e969663badea5d58f43: No bootstrap required, opened a new log
I20260927 02:37:49.419525 21812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:49.412134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:60675 (local address 127.0.0.1:45777)
0927 02:37:49.412544 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:49.412558 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:49.412592 (+ 34us) server_negotiation.cc:407] Connection header received
0927 02:37:49.413247 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:49.413264 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:49.413591 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:49.413876 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:49.414335 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:49.415334 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:49.416196 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:49.416654 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:49.416796 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:49.417739 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:49.417746 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:49.417765 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:37:49.418271 (+ 506us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:49.418345 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:37:49.418377 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:49.418513 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:49.418699 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 02:37:49.418836 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":121,"threads_started":1}
I20260927 02:37:49.422297 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:37:49.423256 21809 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83b9d959cc564e969663badea5d58f43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b9d959cc564e969663badea5d58f43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45777 } }
I20260927 02:37:49.423516 21809 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83b9d959cc564e969663badea5d58f43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:37:49.423574 21809 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83b9d959cc564e969663badea5d58f43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b9d959cc564e969663badea5d58f43, State: Initialized, Role: FOLLOWER
I20260927 02:37:49.424012 21809 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83b9d959cc564e969663badea5d58f43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b9d959cc564e969663badea5d58f43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45777 } }
I20260927 02:37:49.424173 21809 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83b9d959cc564e969663badea5d58f43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:37:49.424247 21809 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83b9d959cc564e969663badea5d58f43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:37:49.424337 21809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83b9d959cc564e969663badea5d58f43 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:37:49.425213 21809 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83b9d959cc564e969663badea5d58f43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b9d959cc564e969663badea5d58f43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45777 } }
I20260927 02:37:49.425518 21809 leader_election.cc:302] T 00000000000000000000000000000000 P 83b9d959cc564e969663badea5d58f43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83b9d959cc564e969663badea5d58f43; no voters:
I20260927 02:37:49.425802 21809 leader_election.cc:288] T 00000000000000000000000000000000 P 83b9d959cc564e969663badea5d58f43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:37:49.425873 21819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83b9d959cc564e969663badea5d58f43 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:37:49.426105 21819 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83b9d959cc564e969663badea5d58f43 [term 1 LEADER]: Becoming Leader. State: Replica: 83b9d959cc564e969663badea5d58f43, State: Running, Role: LEADER
I20260927 02:37:49.426443 21819 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83b9d959cc564e969663badea5d58f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b9d959cc564e969663badea5d58f43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45777 } }
I20260927 02:37:49.426970 21809 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83b9d959cc564e969663badea5d58f43 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:37:49.428190 21820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83b9d959cc564e969663badea5d58f43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83b9d959cc564e969663badea5d58f43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b9d959cc564e969663badea5d58f43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45777 } } }
I20260927 02:37:49.428321 21820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83b9d959cc564e969663badea5d58f43 [sys.catalog]: This master's current role is: LEADER
I20260927 02:37:49.428658 21821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83b9d959cc564e969663badea5d58f43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83b9d959cc564e969663badea5d58f43. Latest consensus state: current_term: 1 leader_uuid: "83b9d959cc564e969663badea5d58f43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83b9d959cc564e969663badea5d58f43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45777 } } }
I20260927 02:37:49.429481 21821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83b9d959cc564e969663badea5d58f43 [sys.catalog]: This master's current role is: LEADER
I20260927 02:37:49.429937 21812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:49.424173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:42429 (local address 127.0.0.1:45777)
0927 02:37:49.424296 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:49.424300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:49.425192 (+ 892us) server_negotiation.cc:407] Connection header received
0927 02:37:49.425318 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:49.425321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:49.425369 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:49.425447 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:49.425717 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:49.426474 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:49.427608 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:49.427942 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:49.428058 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:49.429389 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:49.429394 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:49.429397 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:37:49.429409 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:49.429437 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:37:49.429450 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:49.429536 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:49.429764 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 02:37:49.429802 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260927 02:37:49.431715 21834 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 83b9d959cc564e969663badea5d58f43: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:37:49.431783 21834 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:37:49.431859 21832 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:37:49.432564 21832 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:37:49.437466 21832 catalog_manager.cc:1347] Generated new cluster ID: 761f15f5809a4a23b546a3e12f2dd4ed
I20260927 02:37:49.437527 21832 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:37:49.457726 21832 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:37:49.458922 21832 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:37:49.466490 21832 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83b9d959cc564e969663badea5d58f43: Generated new TSK 0
I20260927 02:37:49.467257 21832 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:37:49.477073 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 21741
W20260927 02:37:49.481501 21726 connection.cc:535] client connection to 127.0.0.1:45777 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45777): Connection reset by peer (error 104)
2026-09-27T02:37:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1299 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476410933508-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:37:49 dist-test-slave-grlk krb5kdc[21840](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:37:49 dist-test-slave-grlk krb5kdc[21840](info): set up 2 sockets
Sep 27 02:37:49 dist-test-slave-grlk krb5kdc[21840](info): commencing operation
krb5kdc: starting...
W20260927 02:37:50.424840 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476410933508-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476410933508-392-0/krb5kdc/test-user.keytab.
Sep 27 02:37:50 dist-test-slave-grlk krb5kdc[21840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:37:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:37:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476410933508-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476410933508-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:37:50.531383 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33017
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476410933508-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:37:50.635968 21853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:37:50.636193 21853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:37:50.636229 21853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:37:50.640041 21853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:37:50.640090 21853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:37:50.640108 21853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:37:50.640122 21853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:37:50.640136 21853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:37:50.645054 21853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33017
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476410933508-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:37:50.645905 21853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:37:50.646788 21853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:37:50.654531 21859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:50.654542 21861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:50.654549 21858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:50.654743 21853 server_base.cc:1034] running on GCE node
I20260927 02:37:50.655318 21853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:37:50.656375 21853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:37:50.657524 21853 hybrid_clock.cc:648] HybridClock initialized: now 1790476670657500 us; error 40 us; skew 500 ppm
Sep 27 02:37:50 dist-test-slave-grlk krb5kdc[21840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:37:50.660043 21853 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:37:50.660811 21853 webserver.cc:460] Webserver started at http://127.0.0.1:43151/ using document root <none> and password file <none>
I20260927 02:37:50.661325 21853 fs_manager.cc:359] Metadata directory not provided
I20260927 02:37:50.661388 21853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:37:50.661571 21853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:37:50.663417 21853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "2d7d5dd689804a34aa9d6bd46ccad9c7"
format_stamp: "Formatted at 2026-09-27 02:37:50 on dist-test-slave-grlk"
server_key: "46a18f70357082c65084536e4d41b3c5"
server_key_iv: "a921f0c14065f9beca9394d4994d058e"
server_key_version: "encryptionkey@0"
I20260927 02:37:50.663910 21853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "2d7d5dd689804a34aa9d6bd46ccad9c7"
format_stamp: "Formatted at 2026-09-27 02:37:50 on dist-test-slave-grlk"
server_key: "46a18f70357082c65084536e4d41b3c5"
server_key_iv: "a921f0c14065f9beca9394d4994d058e"
server_key_version: "encryptionkey@0"
I20260927 02:37:50.667282 21853 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:37:50.669400 21868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:50.670455 21853 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:37:50.670567 21853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "2d7d5dd689804a34aa9d6bd46ccad9c7"
format_stamp: "Formatted at 2026-09-27 02:37:50 on dist-test-slave-grlk"
server_key: "46a18f70357082c65084536e4d41b3c5"
server_key_iv: "a921f0c14065f9beca9394d4994d058e"
server_key_version: "encryptionkey@0"
I20260927 02:37:50.670650 21853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:37:50.674729 21853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:37:50.677232 21853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:37:50.677379 21853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:37:50.683305 21853 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33017
I20260927 02:37:50.683344 21920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33017 every 8 connection(s)
I20260927 02:37:50.684096 21853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:37:50.686146 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 21853
I20260927 02:37:50.686262 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:37:50.686529 392 external_mini_cluster.cc:1428] Setting key 6c8ba55a1f5aa8ec7aae7944676b99ef
I20260927 02:37:50.686821 21921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:37:50.691608 21924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:50.687779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:51959 (local address 127.0.0.1:33017)
0927 02:37:50.688187 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:50.688205 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:50.688245 (+ 40us) server_negotiation.cc:407] Connection header received
0927 02:37:50.689027 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:50.689050 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:50.689413 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:50.689755 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:50.690332 (+ 577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:50.690337 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:50.690351 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:37:50.690361 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:50.690428 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:37:50.690455 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:50.690585 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:50.690653 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 02:37:50.690795 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":121,"threads_started":1}
I20260927 02:37:50.693473 21921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d7d5dd689804a34aa9d6bd46ccad9c7: Bootstrap starting.
I20260927 02:37:50.694164 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:37:50.695668 21921 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d7d5dd689804a34aa9d6bd46ccad9c7: Neither blocks nor log segments found. Creating new log.
I20260927 02:37:50.696426 21921 log.cc:824] T 00000000000000000000000000000000 P 2d7d5dd689804a34aa9d6bd46ccad9c7: Log is configured to *not* fsync() on all Append() calls
I20260927 02:37:50.697309 21924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:50.695907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.21:51937 (local address 127.0.0.1:33017)
0927 02:37:50.696000 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:50.696004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:50.696194 (+ 190us) server_negotiation.cc:407] Connection header received
0927 02:37:50.696287 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:50.696290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:50.696319 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:50.696377 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:50.696980 (+ 603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:50.696983 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:50.696986 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:37:50.696997 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:50.697021 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:37:50.697031 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:50.697097 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:50.697171 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 02:37:50.697207 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 02:37:50.698272 21921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d7d5dd689804a34aa9d6bd46ccad9c7: No bootstrap required, opened a new log
I20260927 02:37:50.701622 21921 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d7d5dd689804a34aa9d6bd46ccad9c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d7d5dd689804a34aa9d6bd46ccad9c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33017 } }
I20260927 02:37:50.701809 21921 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d7d5dd689804a34aa9d6bd46ccad9c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:37:50.701858 21921 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d7d5dd689804a34aa9d6bd46ccad9c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d7d5dd689804a34aa9d6bd46ccad9c7, State: Initialized, Role: FOLLOWER
I20260927 02:37:50.702241 21921 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d7d5dd689804a34aa9d6bd46ccad9c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d7d5dd689804a34aa9d6bd46ccad9c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33017 } }
I20260927 02:37:50.702368 21921 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d7d5dd689804a34aa9d6bd46ccad9c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:37:50.702428 21921 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d7d5dd689804a34aa9d6bd46ccad9c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:37:50.702517 21921 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d7d5dd689804a34aa9d6bd46ccad9c7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:37:50.703238 21921 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d7d5dd689804a34aa9d6bd46ccad9c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d7d5dd689804a34aa9d6bd46ccad9c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33017 } }
I20260927 02:37:50.703522 21921 leader_election.cc:302] T 00000000000000000000000000000000 P 2d7d5dd689804a34aa9d6bd46ccad9c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d7d5dd689804a34aa9d6bd46ccad9c7; no voters:
I20260927 02:37:50.703755 21921 leader_election.cc:288] T 00000000000000000000000000000000 P 2d7d5dd689804a34aa9d6bd46ccad9c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:37:50.703840 21931 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d7d5dd689804a34aa9d6bd46ccad9c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:37:50.704042 21931 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d7d5dd689804a34aa9d6bd46ccad9c7 [term 1 LEADER]: Becoming Leader. State: Replica: 2d7d5dd689804a34aa9d6bd46ccad9c7, State: Running, Role: LEADER
I20260927 02:37:50.704342 21931 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d7d5dd689804a34aa9d6bd46ccad9c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d7d5dd689804a34aa9d6bd46ccad9c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33017 } }
I20260927 02:37:50.704734 21921 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d7d5dd689804a34aa9d6bd46ccad9c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:37:50.705696 21932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d7d5dd689804a34aa9d6bd46ccad9c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d7d5dd689804a34aa9d6bd46ccad9c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d7d5dd689804a34aa9d6bd46ccad9c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33017 } } }
I20260927 02:37:50.705892 21932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d7d5dd689804a34aa9d6bd46ccad9c7 [sys.catalog]: This master's current role is: LEADER
I20260927 02:37:50.705936 21933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d7d5dd689804a34aa9d6bd46ccad9c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d7d5dd689804a34aa9d6bd46ccad9c7. Latest consensus state: current_term: 1 leader_uuid: "2d7d5dd689804a34aa9d6bd46ccad9c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d7d5dd689804a34aa9d6bd46ccad9c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33017 } } }
I20260927 02:37:50.706051 21933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d7d5dd689804a34aa9d6bd46ccad9c7 [sys.catalog]: This master's current role is: LEADER
I20260927 02:37:50.706354 21940 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:37:50.709221 21940 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:37:50.714015 21940 catalog_manager.cc:1347] Generated new cluster ID: da10cf3a266f46329b4e5f35aacc8d9e
I20260927 02:37:50.714075 21940 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:37:50.725319 21940 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:37:50.725668 21940 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:37:50.731608 21940 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d7d5dd689804a34aa9d6bd46ccad9c7: Generated new TSK 0
I20260927 02:37:50.732203 21940 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:37:50.741735 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 21853
2026-09-27T02:37:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1264 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476410933508-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:37:50 dist-test-slave-grlk krb5kdc[21952](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:37:50 dist-test-slave-grlk krb5kdc[21952](info): set up 2 sockets
Sep 27 02:37:50 dist-test-slave-grlk krb5kdc[21952](info): commencing operation
krb5kdc: starting...
W20260927 02:37:51.684767 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476410933508-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476410933508-392-0/krb5kdc/test-user.keytab.
Sep 27 02:37:51 dist-test-slave-grlk krb5kdc[21952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:37:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:37:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:37:51.793390 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42869
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42869
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:37:51.898006 21965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:37:51.898240 21965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:37:51.898294 21965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:37:51.902235 21965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:37:51.902299 21965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:37:51.902324 21965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:37:51.902346 21965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:37:51.902371 21965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:37:51.907487 21965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42869
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:37:51.908494 21965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:37:51.909504 21965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:37:51.918324 21973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:51.918308 21971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:51.918367 21970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:51.919004 21965 server_base.cc:1034] running on GCE node
I20260927 02:37:51.919660 21965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:37:51.920861 21965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:37:51.922035 21965 hybrid_clock.cc:648] HybridClock initialized: now 1790476671922020 us; error 34 us; skew 500 ppm
Sep 27 02:37:51 dist-test-slave-grlk krb5kdc[21952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:37:51.928857 21965 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:37:51.929787 21965 webserver.cc:460] Webserver started at http://127.0.98.62:46741/ using document root <none> and password file <none>
I20260927 02:37:51.930320 21965 fs_manager.cc:359] Metadata directory not provided
I20260927 02:37:51.930383 21965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:37:51.930572 21965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:37:51.932547 21965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "fa568d7fda8c494f93f26cddbb28e539"
format_stamp: "Formatted at 2026-09-27 02:37:51 on dist-test-slave-grlk"
server_key: "d40344a3d84b0252b649e5b2c2793c7b"
server_key_iv: "6f0b51c974d5ade9ec0aa3fea1bc9ba8"
server_key_version: "encryptionkey@0"
I20260927 02:37:51.933099 21965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "fa568d7fda8c494f93f26cddbb28e539"
format_stamp: "Formatted at 2026-09-27 02:37:51 on dist-test-slave-grlk"
server_key: "d40344a3d84b0252b649e5b2c2793c7b"
server_key_iv: "6f0b51c974d5ade9ec0aa3fea1bc9ba8"
server_key_version: "encryptionkey@0"
I20260927 02:37:51.936556 21965 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:37:51.938789 21980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:51.939867 21965 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:37:51.939972 21965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "fa568d7fda8c494f93f26cddbb28e539"
format_stamp: "Formatted at 2026-09-27 02:37:51 on dist-test-slave-grlk"
server_key: "d40344a3d84b0252b649e5b2c2793c7b"
server_key_iv: "6f0b51c974d5ade9ec0aa3fea1bc9ba8"
server_key_version: "encryptionkey@0"
I20260927 02:37:51.940055 21965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:37:51.952260 21965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:37:51.955024 21965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:37:51.955178 21965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:37:51.961664 21965 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42869
I20260927 02:37:51.961694 22032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42869 every 8 connection(s)
I20260927 02:37:51.962531 21965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:37:51.965322 22033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:37:51.968564 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 21965
I20260927 02:37:51.968693 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:37:51.968993 392 external_mini_cluster.cc:1428] Setting key fe296e89f26128789c63cf98e8531651
I20260927 02:37:51.971066 22033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa568d7fda8c494f93f26cddbb28e539: Bootstrap starting.
I20260927 02:37:51.973098 22033 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa568d7fda8c494f93f26cddbb28e539: Neither blocks nor log segments found. Creating new log.
I20260927 02:37:51.973764 22033 log.cc:824] T 00000000000000000000000000000000 P fa568d7fda8c494f93f26cddbb28e539: Log is configured to *not* fsync() on all Append() calls
I20260927 02:37:51.975382 22033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa568d7fda8c494f93f26cddbb28e539: No bootstrap required, opened a new log
Sep 27 02:37:51 dist-test-slave-grlk krb5kdc[21952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:37:51.978814 22033 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa568d7fda8c494f93f26cddbb28e539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa568d7fda8c494f93f26cddbb28e539" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42869 } }
I20260927 02:37:51.979023 22033 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa568d7fda8c494f93f26cddbb28e539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:37:51.979079 22033 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa568d7fda8c494f93f26cddbb28e539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa568d7fda8c494f93f26cddbb28e539, State: Initialized, Role: FOLLOWER
I20260927 02:37:51.979478 22033 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa568d7fda8c494f93f26cddbb28e539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa568d7fda8c494f93f26cddbb28e539" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42869 } }
I20260927 02:37:51.979602 22033 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa568d7fda8c494f93f26cddbb28e539 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:37:51.979663 22033 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa568d7fda8c494f93f26cddbb28e539 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:37:51.979764 22033 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa568d7fda8c494f93f26cddbb28e539 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:37:51.980540 22033 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa568d7fda8c494f93f26cddbb28e539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa568d7fda8c494f93f26cddbb28e539" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42869 } }
I20260927 02:37:51.980876 22033 leader_election.cc:302] T 00000000000000000000000000000000 P fa568d7fda8c494f93f26cddbb28e539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa568d7fda8c494f93f26cddbb28e539; no voters:
I20260927 02:37:51.981201 22033 leader_election.cc:288] T 00000000000000000000000000000000 P fa568d7fda8c494f93f26cddbb28e539 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:37:51.981281 22038 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa568d7fda8c494f93f26cddbb28e539 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:37:51.981473 22038 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa568d7fda8c494f93f26cddbb28e539 [term 1 LEADER]: Becoming Leader. State: Replica: fa568d7fda8c494f93f26cddbb28e539, State: Running, Role: LEADER
I20260927 02:37:51.981817 22038 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa568d7fda8c494f93f26cddbb28e539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa568d7fda8c494f93f26cddbb28e539" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42869 } }
I20260927 02:37:51.982069 22036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:51.970292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47431 (local address 127.0.98.62:42869)
0927 02:37:51.970698 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:51.970711 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:51.970743 (+ 32us) server_negotiation.cc:407] Connection header received
0927 02:37:51.971383 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:51.971399 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:51.971709 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:51.972001 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:51.972755 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:51.973791 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:51.974618 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:51.975059 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:51.975193 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:51.977526 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:51.977531 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:51.977543 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:37:51.977568 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:51.979128 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:51.979589 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:51.979595 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:51.979601 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:51.979730 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:51.980022 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:51.980025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:51.980027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:51.980386 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:51.980579 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:51.980875 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 02:37:51.981119 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":124,"threads_started":1}
I20260927 02:37:51.982388 22033 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa568d7fda8c494f93f26cddbb28e539 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:37:51.983467 22039 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa568d7fda8c494f93f26cddbb28e539 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa568d7fda8c494f93f26cddbb28e539" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa568d7fda8c494f93f26cddbb28e539" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42869 } } }
I20260927 02:37:51.983467 22040 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa568d7fda8c494f93f26cddbb28e539 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa568d7fda8c494f93f26cddbb28e539. Latest consensus state: current_term: 1 leader_uuid: "fa568d7fda8c494f93f26cddbb28e539" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa568d7fda8c494f93f26cddbb28e539" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42869 } } }
I20260927 02:37:51.983610 22039 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa568d7fda8c494f93f26cddbb28e539 [sys.catalog]: This master's current role is: LEADER
I20260927 02:37:51.983610 22040 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa568d7fda8c494f93f26cddbb28e539 [sys.catalog]: This master's current role is: LEADER
I20260927 02:37:51.984100 22047 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:37:51.985216 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:37:51.987345 22047 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:37:51.993037 22047 catalog_manager.cc:1347] Generated new cluster ID: 126805fef00f4382bd52750643fec213
I20260927 02:37:51.993085 22047 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:37:51.995028 22036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:51.987165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35913 (local address 127.0.98.62:42869)
0927 02:37:51.987273 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:51.987277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:51.987467 (+ 190us) server_negotiation.cc:407] Connection header received
0927 02:37:51.987632 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:51.987635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:51.987680 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:51.987765 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:51.988521 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:51.989218 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:51.989988 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:51.990241 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:51.990331 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:51.991693 (+ 1362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:51.991697 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:51.991699 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:37:51.991722 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:51.993342 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:51.993830 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:51.993834 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:51.993836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:51.993917 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:51.994207 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:51.994210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:51.994212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:51.994425 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:51.994566 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:51.994792 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 02:37:51.994886 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 02:37:52.006685 22047 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:37:52.007491 22047 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:37:52.015580 22047 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa568d7fda8c494f93f26cddbb28e539: Generated new TSK 0
I20260927 02:37:52.016342 22047 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:37:52.022459 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 21965
W20260927 02:37:52.026849 21950 connection.cc:535] client connection to 127.0.98.62:42869 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:42869): Connection reset by peer (error 104)
2026-09-27T02:37:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1280 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476410933508-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:37:52 dist-test-slave-grlk krb5kdc[22064](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:37:52 dist-test-slave-grlk krb5kdc[22064](info): set up 2 sockets
Sep 27 02:37:52 dist-test-slave-grlk krb5kdc[22064](info): commencing operation
krb5kdc: starting...
W20260927 02:37:52.968786 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476410933508-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476410933508-392-0/krb5kdc/test-user.keytab.
Sep 27 02:37:53 dist-test-slave-grlk krb5kdc[22064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:37:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:37:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:37:53.080109 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37269
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:37:53.186430 22077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:37:53.186671 22077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:37:53.186724 22077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:37:53.190599 22077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:37:53.190654 22077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:37:53.190680 22077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:37:53.190702 22077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:37:53.190726 22077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:37:53.197880 22077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37269
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:37:53.199298 22077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:37:53.200565 22077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:37:53.209270 22083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:53.209285 22085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:53.209271 22082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:53.209429 22077 server_base.cc:1034] running on GCE node
I20260927 02:37:53.209947 22077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:37:53.210907 22077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:37:53.212051 22077 hybrid_clock.cc:648] HybridClock initialized: now 1790476673212029 us; error 32 us; skew 500 ppm
Sep 27 02:37:53 dist-test-slave-grlk krb5kdc[22064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476673, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:37:53.214608 22077 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:37:53.215397 22077 webserver.cc:460] Webserver started at http://127.0.98.62:34761/ using document root <none> and password file <none>
I20260927 02:37:53.215881 22077 fs_manager.cc:359] Metadata directory not provided
I20260927 02:37:53.215935 22077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:37:53.216082 22077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:37:53.217994 22077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "a97d19bd7d3b4a1ca229639fed95f64a"
format_stamp: "Formatted at 2026-09-27 02:37:53 on dist-test-slave-grlk"
server_key: "da0208345c38049db426181c90f415ff"
server_key_iv: "a33b22993462f1b64c9b3dfdfba4160d"
server_key_version: "encryptionkey@0"
I20260927 02:37:53.218461 22077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "a97d19bd7d3b4a1ca229639fed95f64a"
format_stamp: "Formatted at 2026-09-27 02:37:53 on dist-test-slave-grlk"
server_key: "da0208345c38049db426181c90f415ff"
server_key_iv: "a33b22993462f1b64c9b3dfdfba4160d"
server_key_version: "encryptionkey@0"
I20260927 02:37:53.221666 22077 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:37:53.223762 22092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:53.224792 22077 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 02:37:53.224880 22077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "a97d19bd7d3b4a1ca229639fed95f64a"
format_stamp: "Formatted at 2026-09-27 02:37:53 on dist-test-slave-grlk"
server_key: "da0208345c38049db426181c90f415ff"
server_key_iv: "a33b22993462f1b64c9b3dfdfba4160d"
server_key_version: "encryptionkey@0"
I20260927 02:37:53.225025 22077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:37:53.236217 22077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:37:53.238963 22077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:37:53.239100 22077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:37:53.245275 22077 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37269
I20260927 02:37:53.245290 22144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37269 every 8 connection(s)
I20260927 02:37:53.246141 22077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:37:53.248556 22145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:37:53.253520 22145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a97d19bd7d3b4a1ca229639fed95f64a: Bootstrap starting.
I20260927 02:37:53.255209 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 22077
I20260927 02:37:53.255321 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:37:53.255548 392 external_mini_cluster.cc:1428] Setting key f028221e76122eb79e0c3236bade3fd5
I20260927 02:37:53.255771 22145 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a97d19bd7d3b4a1ca229639fed95f64a: Neither blocks nor log segments found. Creating new log.
I20260927 02:37:53.256548 22145 log.cc:824] T 00000000000000000000000000000000 P a97d19bd7d3b4a1ca229639fed95f64a: Log is configured to *not* fsync() on all Append() calls
I20260927 02:37:53.258240 22145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a97d19bd7d3b4a1ca229639fed95f64a: No bootstrap required, opened a new log
I20260927 02:37:53.262065 22145 raft_consensus.cc:348] T 00000000000000000000000000000000 P a97d19bd7d3b4a1ca229639fed95f64a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a97d19bd7d3b4a1ca229639fed95f64a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37269 } }
I20260927 02:37:53.262269 22145 raft_consensus.cc:374] T 00000000000000000000000000000000 P a97d19bd7d3b4a1ca229639fed95f64a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:37:53.262301 22145 raft_consensus.cc:729] T 00000000000000000000000000000000 P a97d19bd7d3b4a1ca229639fed95f64a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97d19bd7d3b4a1ca229639fed95f64a, State: Initialized, Role: FOLLOWER
I20260927 02:37:53.262722 22145 consensus_queue.cc:260] T 00000000000000000000000000000000 P a97d19bd7d3b4a1ca229639fed95f64a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a97d19bd7d3b4a1ca229639fed95f64a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37269 } }
I20260927 02:37:53.262861 22145 raft_consensus.cc:388] T 00000000000000000000000000000000 P a97d19bd7d3b4a1ca229639fed95f64a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:37:53.262923 22145 raft_consensus.cc:482] T 00000000000000000000000000000000 P a97d19bd7d3b4a1ca229639fed95f64a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:37:53.263024 22145 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a97d19bd7d3b4a1ca229639fed95f64a [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:37:53.263499 22148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:53.256489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59655 (local address 127.0.98.62:37269)
0927 02:37:53.256856 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:53.256871 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:53.256903 (+ 32us) server_negotiation.cc:407] Connection header received
0927 02:37:53.257528 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:53.257543 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:53.257834 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:53.258082 (+ 248us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:53.258403 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:53.259335 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:53.260519 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:53.260991 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:53.261097 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:53.261858 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:53.261863 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:53.261874 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:37:53.262228 (+ 354us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:53.262286 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:37:53.262318 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:53.262437 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:53.262634 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 02:37:53.262744 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":114,"threads_started":1}
I20260927 02:37:53.263844 22145 raft_consensus.cc:504] T 00000000000000000000000000000000 P a97d19bd7d3b4a1ca229639fed95f64a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a97d19bd7d3b4a1ca229639fed95f64a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37269 } }
I20260927 02:37:53.264143 22145 leader_election.cc:302] T 00000000000000000000000000000000 P a97d19bd7d3b4a1ca229639fed95f64a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a97d19bd7d3b4a1ca229639fed95f64a; no voters:
I20260927 02:37:53.264441 22145 leader_election.cc:288] T 00000000000000000000000000000000 P a97d19bd7d3b4a1ca229639fed95f64a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:37:53.264480 22150 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a97d19bd7d3b4a1ca229639fed95f64a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:37:53.264698 22150 raft_consensus.cc:686] T 00000000000000000000000000000000 P a97d19bd7d3b4a1ca229639fed95f64a [term 1 LEADER]: Becoming Leader. State: Replica: a97d19bd7d3b4a1ca229639fed95f64a, State: Running, Role: LEADER
I20260927 02:37:53.265048 22150 consensus_queue.cc:237] T 00000000000000000000000000000000 P a97d19bd7d3b4a1ca229639fed95f64a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a97d19bd7d3b4a1ca229639fed95f64a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37269 } }
I20260927 02:37:53.265556 22145 sys_catalog.cc:564] T 00000000000000000000000000000000 P a97d19bd7d3b4a1ca229639fed95f64a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:37:53.266142 22152 sys_catalog.cc:455] T 00000000000000000000000000000000 P a97d19bd7d3b4a1ca229639fed95f64a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a97d19bd7d3b4a1ca229639fed95f64a. Latest consensus state: current_term: 1 leader_uuid: "a97d19bd7d3b4a1ca229639fed95f64a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a97d19bd7d3b4a1ca229639fed95f64a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37269 } } }
I20260927 02:37:53.266258 22152 sys_catalog.cc:458] T 00000000000000000000000000000000 P a97d19bd7d3b4a1ca229639fed95f64a [sys.catalog]: This master's current role is: LEADER
I20260927 02:37:53.266469 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:37:53.266587 22157 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:37:53.266567 22151 sys_catalog.cc:455] T 00000000000000000000000000000000 P a97d19bd7d3b4a1ca229639fed95f64a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a97d19bd7d3b4a1ca229639fed95f64a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a97d19bd7d3b4a1ca229639fed95f64a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37269 } } }
I20260927 02:37:53.266650 22151 sys_catalog.cc:458] T 00000000000000000000000000000000 P a97d19bd7d3b4a1ca229639fed95f64a [sys.catalog]: This master's current role is: LEADER
I20260927 02:37:53.269423 22157 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:37:53.273718 22148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:53.268506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46007 (local address 127.0.98.62:37269)
0927 02:37:53.268597 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:53.268601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:53.269564 (+ 963us) server_negotiation.cc:407] Connection header received
0927 02:37:53.269734 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:53.269737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:53.269779 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:53.269825 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:53.270215 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:53.270751 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:53.271726 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:53.272082 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:53.272185 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:53.273301 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:53.273304 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:53.273307 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:37:53.273319 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:53.273342 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:37:53.273352 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:53.273417 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:53.273560 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 02:37:53.273588 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 02:37:53.275034 22157 catalog_manager.cc:1347] Generated new cluster ID: 64cea6892fe34738b0a09864eca7996c
I20260927 02:37:53.275080 22157 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:37:53.288422 22157 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:37:53.289299 22157 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:37:53.297883 22157 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a97d19bd7d3b4a1ca229639fed95f64a: Generated new TSK 0
I20260927 02:37:53.298447 22157 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:37:53.302848 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 22077
W20260927 02:37:53.308086 22062 connection.cc:535] client connection to 127.0.98.62:37269 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:37269): Connection reset by peer (error 104)
2026-09-27T02:37:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1281 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476410933508-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:37:53 dist-test-slave-grlk krb5kdc[22176](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:37:53 dist-test-slave-grlk krb5kdc[22176](info): set up 2 sockets
Sep 27 02:37:53 dist-test-slave-grlk krb5kdc[22176](info): commencing operation
krb5kdc: starting...
W20260927 02:37:54.268745 392 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476410933508-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476410933508-392-0/krb5kdc/test-user.keytab.
Sep 27 02:37:54 dist-test-slave-grlk krb5kdc[22176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:37:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:37:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:37:54.380125 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33127
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:37:54.484977 22189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:37:54.485231 22189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:37:54.485287 22189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:37:54.489126 22189 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:37:54.489177 22189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:37:54.489202 22189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:37:54.489223 22189 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:37:54.489243 22189 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:37:54.494266 22189 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33127
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:37:54.495234 22189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:37:54.496188 22189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:37:54.503931 22197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:54.504040 22195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:54.503933 22194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:54.504050 22189 server_base.cc:1034] running on GCE node
I20260927 02:37:54.504596 22189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:37:54.505609 22189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:37:54.506758 22189 hybrid_clock.cc:648] HybridClock initialized: now 1790476674506746 us; error 28 us; skew 500 ppm
Sep 27 02:37:54 dist-test-slave-grlk krb5kdc[22176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:37:54.509182 22189 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:37:54.510015 22189 webserver.cc:460] Webserver started at http://127.0.98.62:44113/ using document root <none> and password file <none>
I20260927 02:37:54.510509 22189 fs_manager.cc:359] Metadata directory not provided
I20260927 02:37:54.510577 22189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:37:54.510767 22189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:37:54.512682 22189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "3966274802e1478eb89180423b820b9d"
format_stamp: "Formatted at 2026-09-27 02:37:54 on dist-test-slave-grlk"
server_key: "d35c76452f99dee99fff2eed7dffe5d6"
server_key_iv: "a48f275bf317ac2969f6b5b6bec618d4"
server_key_version: "encryptionkey@0"
I20260927 02:37:54.513196 22189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "3966274802e1478eb89180423b820b9d"
format_stamp: "Formatted at 2026-09-27 02:37:54 on dist-test-slave-grlk"
server_key: "d35c76452f99dee99fff2eed7dffe5d6"
server_key_iv: "a48f275bf317ac2969f6b5b6bec618d4"
server_key_version: "encryptionkey@0"
I20260927 02:37:54.516528 22189 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:37:54.518689 22204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:54.519754 22189 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:37:54.519865 22189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "3966274802e1478eb89180423b820b9d"
format_stamp: "Formatted at 2026-09-27 02:37:54 on dist-test-slave-grlk"
server_key: "d35c76452f99dee99fff2eed7dffe5d6"
server_key_iv: "a48f275bf317ac2969f6b5b6bec618d4"
server_key_version: "encryptionkey@0"
I20260927 02:37:54.519948 22189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:37:54.524248 22189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:37:54.526886 22189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:37:54.527037 22189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:37:54.533294 22189 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33127
I20260927 02:37:54.533322 22256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33127 every 8 connection(s)
I20260927 02:37:54.534140 22189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:37:54.535148 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 22189
I20260927 02:37:54.535265 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:37:54.535501 392 external_mini_cluster.cc:1428] Setting key f9765c6f05b3f4c3b5d504c757d5cffc
I20260927 02:37:54.536844 22257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:37:54.540205 22260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:54.536716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48201 (local address 127.0.98.62:33127)
0927 02:37:54.537115 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:54.537130 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:54.537164 (+ 34us) server_negotiation.cc:407] Connection header received
0927 02:37:54.537775 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:54.537790 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:54.538054 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:54.538328 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:54.539121 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:54.539126 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:54.539137 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:37:54.539146 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:54.539196 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:37:54.539215 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:54.539312 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:54.539402 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 02:37:54.539533 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":105,"threads_started":1}
I20260927 02:37:54.542752 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:37:54.542830 22257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3966274802e1478eb89180423b820b9d: Bootstrap starting.
I20260927 02:37:54.545338 22257 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3966274802e1478eb89180423b820b9d: Neither blocks nor log segments found. Creating new log.
I20260927 02:37:54.546092 22257 log.cc:824] T 00000000000000000000000000000000 P 3966274802e1478eb89180423b820b9d: Log is configured to *not* fsync() on all Append() calls
I20260927 02:37:54.546269 22260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:54.544828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52605 (local address 127.0.98.62:33127)
0927 02:37:54.544957 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:54.544961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:54.545144 (+ 183us) server_negotiation.cc:407] Connection header received
0927 02:37:54.545246 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:54.545248 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:54.545277 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:54.545334 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:54.545942 (+ 608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:54.545946 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:54.545948 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:37:54.545958 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:54.545983 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:37:54.545992 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:54.546056 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:54.546126 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 02:37:54.546161 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 02:37:54.547897 22257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3966274802e1478eb89180423b820b9d: No bootstrap required, opened a new log
I20260927 02:37:54.551501 22257 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3966274802e1478eb89180423b820b9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3966274802e1478eb89180423b820b9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33127 } }
I20260927 02:37:54.551745 22257 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3966274802e1478eb89180423b820b9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:37:54.551848 22257 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3966274802e1478eb89180423b820b9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3966274802e1478eb89180423b820b9d, State: Initialized, Role: FOLLOWER
I20260927 02:37:54.552258 22257 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3966274802e1478eb89180423b820b9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3966274802e1478eb89180423b820b9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33127 } }
I20260927 02:37:54.552374 22257 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3966274802e1478eb89180423b820b9d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:37:54.552417 22257 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3966274802e1478eb89180423b820b9d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:37:54.552484 22257 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3966274802e1478eb89180423b820b9d [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:37:54.553301 22257 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3966274802e1478eb89180423b820b9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3966274802e1478eb89180423b820b9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33127 } }
I20260927 02:37:54.553580 22257 leader_election.cc:302] T 00000000000000000000000000000000 P 3966274802e1478eb89180423b820b9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3966274802e1478eb89180423b820b9d; no voters:
I20260927 02:37:54.553804 22257 leader_election.cc:288] T 00000000000000000000000000000000 P 3966274802e1478eb89180423b820b9d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:37:54.553900 22267 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3966274802e1478eb89180423b820b9d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:37:54.554083 22267 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3966274802e1478eb89180423b820b9d [term 1 LEADER]: Becoming Leader. State: Replica: 3966274802e1478eb89180423b820b9d, State: Running, Role: LEADER
I20260927 02:37:54.554399 22267 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3966274802e1478eb89180423b820b9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3966274802e1478eb89180423b820b9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33127 } }
I20260927 02:37:54.554860 22257 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3966274802e1478eb89180423b820b9d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:37:54.555823 22269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3966274802e1478eb89180423b820b9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3966274802e1478eb89180423b820b9d. Latest consensus state: current_term: 1 leader_uuid: "3966274802e1478eb89180423b820b9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3966274802e1478eb89180423b820b9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33127 } } }
I20260927 02:37:54.555948 22269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3966274802e1478eb89180423b820b9d [sys.catalog]: This master's current role is: LEADER
I20260927 02:37:54.556252 22268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3966274802e1478eb89180423b820b9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3966274802e1478eb89180423b820b9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3966274802e1478eb89180423b820b9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33127 } } }
I20260927 02:37:54.556324 22276 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:37:54.556334 22268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3966274802e1478eb89180423b820b9d [sys.catalog]: This master's current role is: LEADER
I20260927 02:37:54.559264 22276 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:37:54.564033 22276 catalog_manager.cc:1347] Generated new cluster ID: 946543583c34485dbb2d5e3a063b771a
I20260927 02:37:54.564164 22276 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:37:54.585842 22276 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:37:54.586177 22276 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:37:54.592181 22276 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3966274802e1478eb89180423b820b9d: Generated new TSK 0
I20260927 02:37:54.592903 22276 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:37:54.598251 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 22189
2026-09-27T02:37:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1294 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476410933508-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:37:54 dist-test-slave-grlk krb5kdc[22288](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:37:54 dist-test-slave-grlk krb5kdc[22288](info): set up 2 sockets
Sep 27 02:37:54 dist-test-slave-grlk krb5kdc[22288](info): commencing operation
krb5kdc: starting...
W20260927 02:37:55.544767 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476410933508-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476410933508-392-0/krb5kdc/test-user.keytab.
Sep 27 02:37:55 dist-test-slave-grlk krb5kdc[22288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:37:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:37:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:37:55.653165 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38091
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:37:55.757396 22301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:37:55.757644 22301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:37:55.757684 22301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:37:55.761777 22301 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:37:55.761844 22301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:37:55.761866 22301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:37:55.761888 22301 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:37:55.761905 22301 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:37:55.767184 22301 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:37:55.768101 22301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:37:55.769040 22301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:37:55.777006 22306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:55.777015 22309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:55.776999 22307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:55.777173 22301 server_base.cc:1034] running on GCE node
I20260927 02:37:55.777770 22301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:37:55.778801 22301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:37:55.779951 22301 hybrid_clock.cc:648] HybridClock initialized: now 1790476675779933 us; error 30 us; skew 500 ppm
Sep 27 02:37:55 dist-test-slave-grlk krb5kdc[22288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:37:55.782488 22301 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:37:55.783277 22301 webserver.cc:460] Webserver started at http://127.0.98.62:36269/ using document root <none> and password file <none>
I20260927 02:37:55.783768 22301 fs_manager.cc:359] Metadata directory not provided
I20260927 02:37:55.783833 22301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:37:55.783996 22301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:37:55.786013 22301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "5c7d7223ffd0487397b7aa0628b3486c"
format_stamp: "Formatted at 2026-09-27 02:37:55 on dist-test-slave-grlk"
server_key: "3d8a69a04900633f21c7bf254a03ab16"
server_key_iv: "3c34932d7e9c5cfffdff025e2f6cf65c"
server_key_version: "encryptionkey@0"
I20260927 02:37:55.786494 22301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "5c7d7223ffd0487397b7aa0628b3486c"
format_stamp: "Formatted at 2026-09-27 02:37:55 on dist-test-slave-grlk"
server_key: "3d8a69a04900633f21c7bf254a03ab16"
server_key_iv: "3c34932d7e9c5cfffdff025e2f6cf65c"
server_key_version: "encryptionkey@0"
I20260927 02:37:55.789973 22301 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260927 02:37:55.792141 22316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:55.793289 22301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 02:37:55.793395 22301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "5c7d7223ffd0487397b7aa0628b3486c"
format_stamp: "Formatted at 2026-09-27 02:37:55 on dist-test-slave-grlk"
server_key: "3d8a69a04900633f21c7bf254a03ab16"
server_key_iv: "3c34932d7e9c5cfffdff025e2f6cf65c"
server_key_version: "encryptionkey@0"
I20260927 02:37:55.793483 22301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:37:55.810639 22301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:37:55.813689 22301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:37:55.813862 22301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:37:55.820559 22301 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38091
I20260927 02:37:55.820585 22368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38091 every 8 connection(s)
I20260927 02:37:55.821434 22301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:37:55.823973 22369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:37:55.828635 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 22301
I20260927 02:37:55.828763 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:37:55.829035 392 external_mini_cluster.cc:1428] Setting key 17a0438a632a49150bed950f6029813c
I20260927 02:37:55.829269 22369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c7d7223ffd0487397b7aa0628b3486c: Bootstrap starting.
I20260927 02:37:55.831696 22369 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c7d7223ffd0487397b7aa0628b3486c: Neither blocks nor log segments found. Creating new log.
I20260927 02:37:55.832561 22369 log.cc:824] T 00000000000000000000000000000000 P 5c7d7223ffd0487397b7aa0628b3486c: Log is configured to *not* fsync() on all Append() calls
I20260927 02:37:55.834579 22369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c7d7223ffd0487397b7aa0628b3486c: No bootstrap required, opened a new log
Sep 27 02:37:55 dist-test-slave-grlk krb5kdc[22288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:37:55.838657 22369 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c7d7223ffd0487397b7aa0628b3486c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c7d7223ffd0487397b7aa0628b3486c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38091 } }
I20260927 02:37:55.838929 22369 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c7d7223ffd0487397b7aa0628b3486c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:37:55.838986 22369 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c7d7223ffd0487397b7aa0628b3486c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c7d7223ffd0487397b7aa0628b3486c, State: Initialized, Role: FOLLOWER
I20260927 02:37:55.839497 22369 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c7d7223ffd0487397b7aa0628b3486c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c7d7223ffd0487397b7aa0628b3486c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38091 } }
I20260927 02:37:55.839648 22369 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c7d7223ffd0487397b7aa0628b3486c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:37:55.839709 22369 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c7d7223ffd0487397b7aa0628b3486c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:37:55.839800 22369 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c7d7223ffd0487397b7aa0628b3486c [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:37:55.840767 22369 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c7d7223ffd0487397b7aa0628b3486c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c7d7223ffd0487397b7aa0628b3486c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38091 } }
I20260927 02:37:55.841137 22369 leader_election.cc:302] T 00000000000000000000000000000000 P 5c7d7223ffd0487397b7aa0628b3486c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c7d7223ffd0487397b7aa0628b3486c; no voters:
I20260927 02:37:55.841457 22369 leader_election.cc:288] T 00000000000000000000000000000000 P 5c7d7223ffd0487397b7aa0628b3486c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:37:55.841539 22374 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c7d7223ffd0487397b7aa0628b3486c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:37:55.841770 22374 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c7d7223ffd0487397b7aa0628b3486c [term 1 LEADER]: Becoming Leader. State: Replica: 5c7d7223ffd0487397b7aa0628b3486c, State: Running, Role: LEADER
I20260927 02:37:55.842109 22374 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c7d7223ffd0487397b7aa0628b3486c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c7d7223ffd0487397b7aa0628b3486c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38091 } }
I20260927 02:37:55.842646 22369 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c7d7223ffd0487397b7aa0628b3486c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:37:55.842936 22372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:55.830285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47415 (local address 127.0.98.62:38091)
0927 02:37:55.830670 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:55.830686 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:55.830720 (+ 34us) server_negotiation.cc:407] Connection header received
0927 02:37:55.831350 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:55.831366 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:55.831697 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:55.831997 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:55.832798 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:55.833938 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:55.835067 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:55.835508 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:55.835619 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:55.838158 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:55.838164 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:55.838179 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:37:55.838209 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:55.840010 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:55.840372 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:55.840378 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:55.840383 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:55.840518 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:55.840743 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:55.840747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:55.840749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:55.841297 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:55.841478 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:55.841776 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 02:37:55.842058 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":99,"threads_started":1}
I20260927 02:37:55.843990 22374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c7d7223ffd0487397b7aa0628b3486c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c7d7223ffd0487397b7aa0628b3486c. Latest consensus state: current_term: 1 leader_uuid: "5c7d7223ffd0487397b7aa0628b3486c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c7d7223ffd0487397b7aa0628b3486c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38091 } } }
I20260927 02:37:55.844159 22374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c7d7223ffd0487397b7aa0628b3486c [sys.catalog]: This master's current role is: LEADER
I20260927 02:37:55.844386 22376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c7d7223ffd0487397b7aa0628b3486c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c7d7223ffd0487397b7aa0628b3486c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c7d7223ffd0487397b7aa0628b3486c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38091 } } }
I20260927 02:37:55.844482 22376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c7d7223ffd0487397b7aa0628b3486c [sys.catalog]: This master's current role is: LEADER
I20260927 02:37:55.844574 22388 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:37:55.845845 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:37:55.847865 22388 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:37:55.853438 22388 catalog_manager.cc:1347] Generated new cluster ID: 623f2cd0762249629054769298d734a1
I20260927 02:37:55.853490 22388 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:37:55.856925 22372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:55.847514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52351 (local address 127.0.98.62:38091)
0927 02:37:55.847630 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:55.847634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:55.849870 (+ 2236us) server_negotiation.cc:407] Connection header received
0927 02:37:55.849976 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:55.849979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:55.850022 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:55.850099 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:55.850827 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:55.851436 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:55.852173 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:55.852424 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:55.852505 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:55.853847 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:55.853851 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:55.853854 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:37:55.853878 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:55.855422 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:55.855829 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:55.855832 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:55.855834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:55.855917 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:55.856145 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:55.856148 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:55.856150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:55.856354 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:55.856481 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:55.856668 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 02:37:55.856774 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 02:37:55.870793 22388 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:37:55.871690 22388 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:37:55.881832 22388 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c7d7223ffd0487397b7aa0628b3486c: Generated new TSK 0
I20260927 02:37:55.882526 22388 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:37:55.891582 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 22301
2026-09-27T02:37:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1293 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476410933508-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:37:55 dist-test-slave-grlk krb5kdc[22400](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:37:55 dist-test-slave-grlk krb5kdc[22400](info): set up 2 sockets
Sep 27 02:37:55 dist-test-slave-grlk krb5kdc[22400](info): commencing operation
krb5kdc: starting...
W20260927 02:37:56.837740 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476410933508-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476410933508-392-0/krb5kdc/test-user.keytab.
Sep 27 02:37:56 dist-test-slave-grlk krb5kdc[22400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:37:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:37:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:37:56.950062 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38645
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:37:57.056900 22413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:37:57.057220 22413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:37:57.057264 22413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:37:57.061266 22413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:37:57.061331 22413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:37:57.061348 22413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:37:57.061363 22413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:37:57.061378 22413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:37:57.066622 22413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:37:57.067593 22413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:37:57.068524 22413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:37:57.076836 22418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:57.076866 22421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:57.076901 22419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:57.077129 22413 server_base.cc:1034] running on GCE node
I20260927 02:37:57.077672 22413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:37:57.078727 22413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:37:57.079887 22413 hybrid_clock.cc:648] HybridClock initialized: now 1790476677079867 us; error 31 us; skew 500 ppm
Sep 27 02:37:57 dist-test-slave-grlk krb5kdc[22400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476677, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:37:57.090390 22413 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:37:57.091337 22413 webserver.cc:460] Webserver started at http://127.0.98.62:38097/ using document root <none> and password file <none>
I20260927 02:37:57.092029 22413 fs_manager.cc:359] Metadata directory not provided
I20260927 02:37:57.092108 22413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:37:57.092326 22413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:37:57.094827 22413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "53511a32bca3404e95e8e7c2e54f5973"
format_stamp: "Formatted at 2026-09-27 02:37:57 on dist-test-slave-grlk"
server_key: "484b984f3a28828541977932f19adce7"
server_key_iv: "bf418f98ea61f0035fe4ccde656f9c60"
server_key_version: "encryptionkey@0"
I20260927 02:37:57.095366 22413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "53511a32bca3404e95e8e7c2e54f5973"
format_stamp: "Formatted at 2026-09-27 02:37:57 on dist-test-slave-grlk"
server_key: "484b984f3a28828541977932f19adce7"
server_key_iv: "bf418f98ea61f0035fe4ccde656f9c60"
server_key_version: "encryptionkey@0"
I20260927 02:37:57.098970 22413 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:37:57.101300 22428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:57.102468 22413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:37:57.102583 22413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "53511a32bca3404e95e8e7c2e54f5973"
format_stamp: "Formatted at 2026-09-27 02:37:57 on dist-test-slave-grlk"
server_key: "484b984f3a28828541977932f19adce7"
server_key_iv: "bf418f98ea61f0035fe4ccde656f9c60"
server_key_version: "encryptionkey@0"
I20260927 02:37:57.102684 22413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:37:57.147228 22413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:37:57.150301 22413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:37:57.150473 22413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:37:57.156785 22413 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38645
I20260927 02:37:57.156872 22480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38645 every 8 connection(s)
I20260927 02:37:57.157721 22413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:37:57.160315 22481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:37:57.165805 22481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53511a32bca3404e95e8e7c2e54f5973: Bootstrap starting.
I20260927 02:37:57.166926 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 22413
I20260927 02:37:57.167057 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:37:57.167307 392 external_mini_cluster.cc:1428] Setting key 6261b2651002a8af6bbd5318dbb0f6cd
I20260927 02:37:57.168339 22481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53511a32bca3404e95e8e7c2e54f5973: Neither blocks nor log segments found. Creating new log.
I20260927 02:37:57.169059 22481 log.cc:824] T 00000000000000000000000000000000 P 53511a32bca3404e95e8e7c2e54f5973: Log is configured to *not* fsync() on all Append() calls
I20260927 02:37:57.170881 22481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53511a32bca3404e95e8e7c2e54f5973: No bootstrap required, opened a new log
I20260927 02:37:57.174717 22481 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53511a32bca3404e95e8e7c2e54f5973 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53511a32bca3404e95e8e7c2e54f5973" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38645 } }
I20260927 02:37:57.174926 22481 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53511a32bca3404e95e8e7c2e54f5973 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:37:57.174980 22481 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53511a32bca3404e95e8e7c2e54f5973 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53511a32bca3404e95e8e7c2e54f5973, State: Initialized, Role: FOLLOWER
I20260927 02:37:57.175366 22481 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53511a32bca3404e95e8e7c2e54f5973 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53511a32bca3404e95e8e7c2e54f5973" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38645 } }
I20260927 02:37:57.175453 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:57.168310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45531 (local address 127.0.98.62:38645)
0927 02:37:57.168724 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:57.168742 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:57.168777 (+ 35us) server_negotiation.cc:407] Connection header received
0927 02:37:57.169456 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:57.169473 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:57.169803 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:57.170068 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:57.170373 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:57.171319 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:57.172450 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:57.172893 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:57.173037 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:57.173803 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:57.173808 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:57.173820 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:37:57.174186 (+ 366us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:57.174234 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:37:57.174256 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:57.174323 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:57.174462 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 02:37:57.174649 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":131,"threads_started":1}
I20260927 02:37:57.175477 22481 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53511a32bca3404e95e8e7c2e54f5973 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:37:57.175554 22481 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53511a32bca3404e95e8e7c2e54f5973 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:37:57.175642 22481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53511a32bca3404e95e8e7c2e54f5973 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:37:57.176496 22481 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53511a32bca3404e95e8e7c2e54f5973 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53511a32bca3404e95e8e7c2e54f5973" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38645 } }
I20260927 02:37:57.176848 22481 leader_election.cc:302] T 00000000000000000000000000000000 P 53511a32bca3404e95e8e7c2e54f5973 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53511a32bca3404e95e8e7c2e54f5973; no voters:
I20260927 02:37:57.177174 22481 leader_election.cc:288] T 00000000000000000000000000000000 P 53511a32bca3404e95e8e7c2e54f5973 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:37:57.177277 22486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53511a32bca3404e95e8e7c2e54f5973 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:37:57.177479 22486 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53511a32bca3404e95e8e7c2e54f5973 [term 1 LEADER]: Becoming Leader. State: Replica: 53511a32bca3404e95e8e7c2e54f5973, State: Running, Role: LEADER
I20260927 02:37:57.177829 22486 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53511a32bca3404e95e8e7c2e54f5973 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53511a32bca3404e95e8e7c2e54f5973" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38645 } }
I20260927 02:37:57.178390 22481 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53511a32bca3404e95e8e7c2e54f5973 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:37:57.178400 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:37:57.179422 22488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53511a32bca3404e95e8e7c2e54f5973 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53511a32bca3404e95e8e7c2e54f5973. Latest consensus state: current_term: 1 leader_uuid: "53511a32bca3404e95e8e7c2e54f5973" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53511a32bca3404e95e8e7c2e54f5973" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38645 } } }
I20260927 02:37:57.179559 22488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53511a32bca3404e95e8e7c2e54f5973 [sys.catalog]: This master's current role is: LEADER
I20260927 02:37:57.179996 22487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53511a32bca3404e95e8e7c2e54f5973 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53511a32bca3404e95e8e7c2e54f5973" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53511a32bca3404e95e8e7c2e54f5973" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38645 } } }
I20260927 02:37:57.180128 22487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53511a32bca3404e95e8e7c2e54f5973 [sys.catalog]: This master's current role is: LEADER
I20260927 02:37:57.181128 22499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:37:57.183858 22499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:37:57.185003 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:57.180698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37205 (local address 127.0.98.62:38645)
0927 02:37:57.180784 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:57.180788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:57.181008 (+ 220us) server_negotiation.cc:407] Connection header received
0927 02:37:57.181108 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:57.181111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:57.181152 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:57.181216 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:57.181531 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:57.182237 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:57.183327 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:57.183637 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:57.183745 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:57.184560 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:57.184564 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:57.184567 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:37:57.184580 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:57.184607 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:37:57.184620 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:57.184689 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:57.184808 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 02:37:57.184859 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 02:37:57.189242 22499 catalog_manager.cc:1347] Generated new cluster ID: 7d897be2c8f8454091c2a3f487cee77a
I20260927 02:37:57.189306 22499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:37:57.204202 22499 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:37:57.205152 22499 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:37:57.210253 22499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53511a32bca3404e95e8e7c2e54f5973: Generated new TSK 0
I20260927 02:37:57.210986 22499 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:37:57.222543 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 22413
2026-09-27T02:37:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1331 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476410933508-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:37:57 dist-test-slave-grlk krb5kdc[22512](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:37:57 dist-test-slave-grlk krb5kdc[22512](info): set up 2 sockets
Sep 27 02:37:57 dist-test-slave-grlk krb5kdc[22512](info): commencing operation
krb5kdc: starting...
W20260927 02:37:58.172793 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476410933508-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476410933508-392-0/krb5kdc/test-user.keytab.
Sep 27 02:37:58 dist-test-slave-grlk krb5kdc[22512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:37:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:37:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:37:58.285087 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46519
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:37:58.388008 22525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:37:58.388257 22525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:37:58.388309 22525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:37:58.392199 22525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:37:58.392256 22525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:37:58.392282 22525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:37:58.392302 22525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:37:58.392328 22525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:37:58.397439 22525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:37:58.398430 22525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:37:58.399386 22525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:37:58.407074 22530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:58.407090 22533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:58.407203 22531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:58.407289 22525 server_base.cc:1034] running on GCE node
I20260927 02:37:58.407838 22525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:37:58.408877 22525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:37:58.410043 22525 hybrid_clock.cc:648] HybridClock initialized: now 1790476678410020 us; error 33 us; skew 500 ppm
Sep 27 02:37:58 dist-test-slave-grlk krb5kdc[22512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476678, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:37:58.412427 22525 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:37:58.413242 22525 webserver.cc:460] Webserver started at http://127.0.98.62:33767/ using document root <none> and password file <none>
I20260927 02:37:58.413739 22525 fs_manager.cc:359] Metadata directory not provided
I20260927 02:37:58.413812 22525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:37:58.414001 22525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:37:58.415868 22525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "11bb26fcc0ff4c6bb71e0dbd0cfb100f"
format_stamp: "Formatted at 2026-09-27 02:37:58 on dist-test-slave-grlk"
server_key: "d42f0a1e1e6b5a011be93c2fbd8cd24f"
server_key_iv: "bb38cadf821ef9b8842eb3716a4c7cbb"
server_key_version: "encryptionkey@0"
I20260927 02:37:58.416364 22525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "11bb26fcc0ff4c6bb71e0dbd0cfb100f"
format_stamp: "Formatted at 2026-09-27 02:37:58 on dist-test-slave-grlk"
server_key: "d42f0a1e1e6b5a011be93c2fbd8cd24f"
server_key_iv: "bb38cadf821ef9b8842eb3716a4c7cbb"
server_key_version: "encryptionkey@0"
I20260927 02:37:58.419747 22525 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 02:37:58.422122 22540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:58.423234 22525 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:37:58.423348 22525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "11bb26fcc0ff4c6bb71e0dbd0cfb100f"
format_stamp: "Formatted at 2026-09-27 02:37:58 on dist-test-slave-grlk"
server_key: "d42f0a1e1e6b5a011be93c2fbd8cd24f"
server_key_iv: "bb38cadf821ef9b8842eb3716a4c7cbb"
server_key_version: "encryptionkey@0"
I20260927 02:37:58.423447 22525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:37:58.427932 22525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:37:58.430665 22525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:37:58.430836 22525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:37:58.437637 22525 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46519
I20260927 02:37:58.437678 22592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46519 every 8 connection(s)
I20260927 02:37:58.438499 22525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:37:58.440021 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 22525
I20260927 02:37:58.440148 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:37:58.440433 392 external_mini_cluster.cc:1428] Setting key fe0520343441702b31c3160597a6f865
I20260927 02:37:58.441548 22593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:37:58.445734 22596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:58.441740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42355 (local address 127.0.98.62:46519)
0927 02:37:58.442134 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:58.442152 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:58.442192 (+ 40us) server_negotiation.cc:407] Connection header received
0927 02:37:58.442999 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:58.443020 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:58.443351 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:58.443699 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:58.444378 (+ 679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:58.444383 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:58.444397 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:37:58.444408 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:58.444466 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:37:58.444493 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:58.444631 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:58.444726 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 02:37:58.444868 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":109,"threads_started":1}
I20260927 02:37:58.447937 22593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11bb26fcc0ff4c6bb71e0dbd0cfb100f: Bootstrap starting.
I20260927 02:37:58.448210 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:37:58.450083 22593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11bb26fcc0ff4c6bb71e0dbd0cfb100f: Neither blocks nor log segments found. Creating new log.
I20260927 02:37:58.450892 22593 log.cc:824] T 00000000000000000000000000000000 P 11bb26fcc0ff4c6bb71e0dbd0cfb100f: Log is configured to *not* fsync() on all Append() calls
I20260927 02:37:58.451483 22596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:58.449971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53349 (local address 127.0.98.62:46519)
0927 02:37:58.450079 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:58.450083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:58.450326 (+ 243us) server_negotiation.cc:407] Connection header received
0927 02:37:58.450425 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:58.450427 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:58.450455 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:58.450513 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:58.451115 (+ 602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:58.451118 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:58.451120 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:37:58.451129 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:58.451163 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:37:58.451173 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:58.451242 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:58.451339 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 02:37:58.451374 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 02:37:58.452806 22593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11bb26fcc0ff4c6bb71e0dbd0cfb100f: No bootstrap required, opened a new log
I20260927 02:37:58.456112 22593 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11bb26fcc0ff4c6bb71e0dbd0cfb100f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11bb26fcc0ff4c6bb71e0dbd0cfb100f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46519 } }
I20260927 02:37:58.456321 22593 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11bb26fcc0ff4c6bb71e0dbd0cfb100f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:37:58.456372 22593 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11bb26fcc0ff4c6bb71e0dbd0cfb100f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11bb26fcc0ff4c6bb71e0dbd0cfb100f, State: Initialized, Role: FOLLOWER
I20260927 02:37:58.456795 22593 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11bb26fcc0ff4c6bb71e0dbd0cfb100f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11bb26fcc0ff4c6bb71e0dbd0cfb100f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46519 } }
I20260927 02:37:58.456965 22593 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11bb26fcc0ff4c6bb71e0dbd0cfb100f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:37:58.457031 22593 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11bb26fcc0ff4c6bb71e0dbd0cfb100f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:37:58.457136 22593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11bb26fcc0ff4c6bb71e0dbd0cfb100f [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:37:58.457947 22593 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11bb26fcc0ff4c6bb71e0dbd0cfb100f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11bb26fcc0ff4c6bb71e0dbd0cfb100f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46519 } }
I20260927 02:37:58.458252 22593 leader_election.cc:302] T 00000000000000000000000000000000 P 11bb26fcc0ff4c6bb71e0dbd0cfb100f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11bb26fcc0ff4c6bb71e0dbd0cfb100f; no voters:
I20260927 02:37:58.458505 22593 leader_election.cc:288] T 00000000000000000000000000000000 P 11bb26fcc0ff4c6bb71e0dbd0cfb100f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:37:58.458621 22603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11bb26fcc0ff4c6bb71e0dbd0cfb100f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:37:58.458806 22603 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11bb26fcc0ff4c6bb71e0dbd0cfb100f [term 1 LEADER]: Becoming Leader. State: Replica: 11bb26fcc0ff4c6bb71e0dbd0cfb100f, State: Running, Role: LEADER
I20260927 02:37:58.459129 22603 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11bb26fcc0ff4c6bb71e0dbd0cfb100f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11bb26fcc0ff4c6bb71e0dbd0cfb100f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46519 } }
I20260927 02:37:58.459558 22593 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11bb26fcc0ff4c6bb71e0dbd0cfb100f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:37:58.460608 22605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11bb26fcc0ff4c6bb71e0dbd0cfb100f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11bb26fcc0ff4c6bb71e0dbd0cfb100f. Latest consensus state: current_term: 1 leader_uuid: "11bb26fcc0ff4c6bb71e0dbd0cfb100f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11bb26fcc0ff4c6bb71e0dbd0cfb100f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46519 } } }
I20260927 02:37:58.460656 22604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11bb26fcc0ff4c6bb71e0dbd0cfb100f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11bb26fcc0ff4c6bb71e0dbd0cfb100f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11bb26fcc0ff4c6bb71e0dbd0cfb100f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46519 } } }
I20260927 02:37:58.460721 22605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11bb26fcc0ff4c6bb71e0dbd0cfb100f [sys.catalog]: This master's current role is: LEADER
I20260927 02:37:58.460747 22604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11bb26fcc0ff4c6bb71e0dbd0cfb100f [sys.catalog]: This master's current role is: LEADER
I20260927 02:37:58.461050 22612 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:37:58.463986 22612 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:37:58.469082 22612 catalog_manager.cc:1347] Generated new cluster ID: 8215e7aae4b04517a9cc7ff8551fa0ed
I20260927 02:37:58.469156 22612 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:37:58.484444 22612 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:37:58.484760 22612 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:37:58.493587 22612 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11bb26fcc0ff4c6bb71e0dbd0cfb100f: Generated new TSK 0
I20260927 02:37:58.494411 22612 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:37:58.502081 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 22525
2026-09-27T02:37:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1279 ms)
[----------] 18 tests from AuthTokenIssuingTest (23240 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:37:58 dist-test-slave-grlk krb5kdc[22624](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:37:58 dist-test-slave-grlk krb5kdc[22624](info): set up 2 sockets
Sep 27 02:37:58 dist-test-slave-grlk krb5kdc[22624](info): commencing operation
krb5kdc: starting...
W20260927 02:37:59.456959 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/krb5kdc/test-user.keytab.
Sep 27 02:37:59 dist-test-slave-grlk krb5kdc[22624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:37:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:37:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:37:59.567500 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44429
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44429,127.0.98.61:32887,127.0.98.60:32819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:37:59.671212 22637 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 02:37:59.671468 22637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:37:59.671490 22637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:37:59.671527 22637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:37:59.675444 22637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:37:59.675493 22637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:37:59.675511 22637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:37:59.675526 22637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:37:59.675539 22637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:37:59.680534 22637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44429,127.0.98.61:32887,127.0.98.60:32819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:37:59.681466 22637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:37:59.682433 22637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:37:59.690186 22642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:59.690196 22645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:59.690196 22643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:59.690389 22637 server_base.cc:1034] running on GCE node
I20260927 02:37:59.690914 22637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:37:59.691885 22637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:37:59.692999 22637 hybrid_clock.cc:648] HybridClock initialized: now 1790476679692986 us; error 23 us; skew 500 ppm
Sep 27 02:37:59 dist-test-slave-grlk krb5kdc[22624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:37:59.695394 22637 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:37:59.696134 22637 webserver.cc:460] Webserver started at http://127.0.98.62:36943/ using document root <none> and password file <none>
I20260927 02:37:59.696604 22637 fs_manager.cc:359] Metadata directory not provided
I20260927 02:37:59.696656 22637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:37:59.696805 22637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:37:59.698709 22637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "c7836ed54a054c61acf1fa501075cf72"
format_stamp: "Formatted at 2026-09-27 02:37:59 on dist-test-slave-grlk"
server_key: "ff3c7b62ace435352a6ee75c8e30e656"
server_key_iv: "cfc1ea8480df0f4af78130f4ceec1e4a"
server_key_version: "encryptionkey@0"
I20260927 02:37:59.699179 22637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "c7836ed54a054c61acf1fa501075cf72"
format_stamp: "Formatted at 2026-09-27 02:37:59 on dist-test-slave-grlk"
server_key: "ff3c7b62ace435352a6ee75c8e30e656"
server_key_iv: "cfc1ea8480df0f4af78130f4ceec1e4a"
server_key_version: "encryptionkey@0"
I20260927 02:37:59.702579 22637 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:37:59.704846 22652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:59.705981 22637 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:37:59.706077 22637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "c7836ed54a054c61acf1fa501075cf72"
format_stamp: "Formatted at 2026-09-27 02:37:59 on dist-test-slave-grlk"
server_key: "ff3c7b62ace435352a6ee75c8e30e656"
server_key_iv: "cfc1ea8480df0f4af78130f4ceec1e4a"
server_key_version: "encryptionkey@0"
I20260927 02:37:59.706156 22637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:37:59.723899 22637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:37:59.726640 22637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:37:59.726789 22637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:37:59.733043 22637 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44429
I20260927 02:37:59.733314 22704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44429 every 8 connection(s)
I20260927 02:37:59.733990 22637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:37:59.736596 22705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:37:59.740180 22705 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:37:59.742739 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 22637
I20260927 02:37:59.742877 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:37:59.743155 392 external_mini_cluster.cc:1428] Setting key d516514886ce1f1f0044cd76a41acc7c
Sep 27 02:37:59 dist-test-slave-grlk krb5kdc[22624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260927 02:37:59.754436 22707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:59.742261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36110 (local address 127.0.98.62:44429)
0927 02:37:59.743214 (+ 953us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:59.743224 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:59.743266 (+ 42us) server_negotiation.cc:407] Connection header received
0927 02:37:59.743899 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:59.743905 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:59.744134 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:59.744306 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:59.745228 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:59.746085 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:59.747397 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:59.747743 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:59.747852 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:59.750228 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:59.750235 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:59.750248 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:37:59.750273 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:59.751725 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:59.752122 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:59.752126 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:59.752127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:59.752218 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:59.752443 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:59.752446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:59.752447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:59.752801 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:59.752901 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:59.753264 (+ 363us) server_negotiation.cc:299] Negotiation successful
0927 02:37:59.753459 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":834,"thread_start_us":111,"threads_started":1}
I20260927 02:37:59.754436 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:59.742261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44429 (local address 127.0.0.1:36110)
0927 02:37:59.742719 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:59.742758 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:59.743599 (+ 841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:59.744311 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:59.744314 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:59.744710 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:37:59.745089 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:59.745106 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:59.746197 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:59.746201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:59.747291 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:59.747302 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:59.747891 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:59.747895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:59.748026 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:59.748557 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:37:59.748577 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:37:59.750087 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:37:59.751810 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:37:59.751817 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:37:59.751824 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:37:59.752031 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:37:59.752280 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:37:59.752282 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:37:59.752283 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:37:59.752377 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:37:59.752882 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:37:59.752886 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:37:59.753024 (+ 138us) client_negotiation.cc:769] Sending connection context
0927 02:37:59.753193 (+ 169us) client_negotiation.cc:240] Negotiation successful
0927 02:37:59.753539 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":82,"spinlock_wait_cycles":51072,"thread_start_us":129,"threads_started":1}
I20260927 02:37:59.756732 22705 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32887 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:37:59.757483 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:59.757237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32887 (local address 127.0.0.1:35810)
0927 02:37:59.757349 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:59.757417 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 02:37:59.757674 22656 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32887: connect: Connection refused (error 111)
I20260927 02:37:59.758385 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:59.758199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32887 (local address 127.0.0.1:35816)
0927 02:37:59.758294 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:59.758334 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 02:37:59.758498 22705 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:32887: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32887: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 02:37:59.765522 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:32887
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44429,127.0.98.61:32887,127.0.98.60:32819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:37:59.807736 22705 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32887 } attempt: 1
I20260927 02:37:59.808552 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:59.808270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32887 (local address 127.0.0.1:35820)
0927 02:37:59.808415 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:59.808473 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 02:37:59.809252 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:59.809047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32887 (local address 127.0.0.1:35836)
0927 02:37:59.809152 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:59.809195 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 02:37:59.809357 22705 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:32887: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32887: connect: Connection refused (error 111)
W20260927 02:37:59.869549 22712 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 02:37:59.869874 22712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:37:59.869915 22712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:37:59.869979 22712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:37:59.873965 22712 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:37:59.874022 22712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:37:59.874056 22712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:37:59.874086 22712 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:37:59.874115 22712 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:37:59.877527 22705 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32887 } attempt: 2
I20260927 02:37:59.878305 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:59.878029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32887 (local address 127.0.0.1:35850)
0927 02:37:59.878173 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:59.878233 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 02:37:59.878964 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:59.878758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32887 (local address 127.0.0.1:35864)
0927 02:37:59.878865 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:59.878908 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 02:37:59.879089 22705 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:32887: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32887: connect: Connection refused (error 111)
I20260927 02:37:59.879458 22712 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44429,127.0.98.61:32887,127.0.98.60:32819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:32887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:37:59.880447 22712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:37:59.881408 22712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:37:59.889123 22717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:37:59.889101 22720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:59.889201 22712 server_base.cc:1034] running on GCE node
W20260927 02:37:59.889127 22718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:37:59.889837 22712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:37:59.890817 22712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:37:59.891995 22712 hybrid_clock.cc:648] HybridClock initialized: now 1790476679891976 us; error 27 us; skew 500 ppm
Sep 27 02:37:59 dist-test-slave-grlk krb5kdc[22624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:37:59.894567 22712 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 02:37:59.895330 22712 webserver.cc:460] Webserver started at http://127.0.98.61:36681/ using document root <none> and password file <none>
I20260927 02:37:59.895813 22712 fs_manager.cc:359] Metadata directory not provided
I20260927 02:37:59.895867 22712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:37:59.896024 22712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:37:59.897946 22712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/minicluster-data/master-1/data/instance:
uuid: "91118d43f3224924902d724c521d7a99"
format_stamp: "Formatted at 2026-09-27 02:37:59 on dist-test-slave-grlk"
server_key: "07b1ad4ea151fc10d55459feb1a660bc"
server_key_iv: "5dd48e36d2954b0d631ee71dd95be4e4"
server_key_version: "encryptionkey@0"
I20260927 02:37:59.898413 22712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/minicluster-data/master-1/wal/instance:
uuid: "91118d43f3224924902d724c521d7a99"
format_stamp: "Formatted at 2026-09-27 02:37:59 on dist-test-slave-grlk"
server_key: "07b1ad4ea151fc10d55459feb1a660bc"
server_key_iv: "5dd48e36d2954b0d631ee71dd95be4e4"
server_key_version: "encryptionkey@0"
I20260927 02:37:59.901778 22712 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:37:59.903955 22727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:37:59.905103 22712 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:37:59.905203 22712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/minicluster-data/master-1/wal
uuid: "91118d43f3224924902d724c521d7a99"
format_stamp: "Formatted at 2026-09-27 02:37:59 on dist-test-slave-grlk"
server_key: "07b1ad4ea151fc10d55459feb1a660bc"
server_key_iv: "5dd48e36d2954b0d631ee71dd95be4e4"
server_key_version: "encryptionkey@0"
I20260927 02:37:59.905293 22712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:37:59.925791 22712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:37:59.928676 22712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:37:59.928822 22712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:37:59.935145 22712 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:32887
I20260927 02:37:59.935386 22779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:32887 every 8 connection(s)
I20260927 02:37:59.936012 22712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/minicluster-data/master-1/data/info.pb
I20260927 02:37:59.938669 22780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:37:59.940699 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 22712
I20260927 02:37:59.940827 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/minicluster-data/master-1/wal/instance
I20260927 02:37:59.941123 392 external_mini_cluster.cc:1428] Setting key 2d9b87648b7bd63aff7e73d49b8c4a96
I20260927 02:37:59.942829 22780 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 02:37:59 dist-test-slave-grlk krb5kdc[22624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:37:59.955904 22707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:59.944638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36124 (local address 127.0.98.62:44429)
0927 02:37:59.944806 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:59.944810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:59.945680 (+ 870us) server_negotiation.cc:407] Connection header received
0927 02:37:59.946734 (+ 1054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:59.946738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:59.946796 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:59.946885 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:59.947979 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:59.948688 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:59.949937 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:59.950173 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:59.950251 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:59.952594 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:59.952598 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:59.952601 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:37:59.952631 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:59.953902 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:59.954455 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:59.954458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:59.954459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:59.954522 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:59.954863 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:59.954865 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:59.954866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:59.955018 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:59.955094 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:59.955678 (+ 584us) server_negotiation.cc:299] Negotiation successful
0927 02:37:59.955782 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 02:37:59.957055 22783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:59.945040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44429 (local address 127.0.0.1:36124)
0927 02:37:59.945507 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:59.945557 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:59.946515 (+ 958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:59.947045 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:59.947052 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:59.947373 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:37:59.947831 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:59.947844 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:59.948793 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:59.948796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:59.949781 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:59.949792 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:59.950304 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:59.950308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:59.950459 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:59.951022 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:37:59.951044 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:37:59.952435 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:37:59.954006 (+ 1571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:37:59.954012 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:37:59.954027 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:37:59.954368 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:37:59.954596 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:37:59.954600 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:37:59.954602 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:37:59.954780 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:37:59.955118 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:37:59.955125 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:37:59.955441 (+ 316us) client_negotiation.cc:769] Sending connection context
0927 02:37:59.955650 (+ 209us) client_negotiation.cc:240] Negotiation successful
0927 02:37:59.955990 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":112,"threads_started":1}
I20260927 02:37:59.958333 22780 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32887 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Sep 27 02:37:59 dist-test-slave-grlk krb5kdc[22624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 02:37:59.963871 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:32819
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44429,127.0.98.61:32887,127.0.98.60:32819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:37:59.966465 22783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:59.958802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32887 (local address 127.0.0.1:35870)
0927 02:37:59.958918 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:59.958932 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:59.959007 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:59.959526 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:59.959528 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:59.959647 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:37:59.959730 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:59.959735 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:59.960697 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:59.960700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:59.961335 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:59.961346 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:59.961905 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:59.961907 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:59.961976 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:59.962370 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:37:59.962381 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:37:59.963513 (+ 1132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:37:59.965148 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:37:59.965151 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:37:59.965154 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:37:59.965380 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:37:59.965623 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:37:59.965626 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:37:59.965628 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:37:59.965720 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:37:59.966097 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:37:59.966099 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:37:59.966173 (+ 74us) client_negotiation.cc:769] Sending connection context
0927 02:37:59.966259 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 02:37:59.966356 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 02:37:59.966799 22785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:59.958903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35870 (local address 127.0.98.61:32887)
0927 02:37:59.959188 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:59.959196 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:59.959214 (+ 18us) server_negotiation.cc:407] Connection header received
0927 02:37:59.959252 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:59.959258 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:59.959406 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:59.959505 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:59.959812 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:59.960616 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:59.961430 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:59.961820 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:59.961898 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:59.963644 (+ 1746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:59.963652 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:59.963662 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:37:59.963681 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:59.965063 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:59.965471 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:59.965475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:59.965476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:59.965551 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:59.965790 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:59.965792 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:59.965793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:59.966004 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:59.966110 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:59.966347 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 02:37:59.966484 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":138,"threads_started":1}
I20260927 02:37:59.968532 22780 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:37:59.969249 22783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:59.969010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32819 (local address 127.0.0.1:50918)
0927 02:37:59.969125 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:59.969191 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 02:37:59.969429 22728 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32819: connect: Connection refused (error 111)
I20260927 02:37:59.970256 22783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:59.970051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32819 (local address 127.0.0.1:50930)
0927 02:37:59.970152 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:59.970195 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 02:37:59.970232 22705 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32887 } attempt: 3
W20260927 02:37:59.970419 22780 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32819: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32819: connect: Connection refused (error 111)
Sep 27 02:37:59 dist-test-slave-grlk krb5kdc[22624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 02:37:59.978194 22785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:59.970649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35886 (local address 127.0.98.61:32887)
0927 02:37:59.970741 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0927 02:37:59.970745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:37:59.970892 (+ 147us) server_negotiation.cc:407] Connection header received
0927 02:37:59.971011 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:37:59.971013 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:37:59.971061 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:37:59.971122 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:37:59.971617 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:59.972123 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:59.973216 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:37:59.973441 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:37:59.973509 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:59.975428 (+ 1919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:37:59.975431 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:37:59.975432 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:37:59.975452 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:37:59.976695 (+ 1243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:59.977129 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:59.977133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:59.977134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:59.977201 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:37:59.977456 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:37:59.977458 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:37:59.977459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:37:59.977623 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:37:59.977704 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 02:37:59.977960 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 02:37:59.978058 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 02:37:59.978199 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:59.970648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32887 (local address 127.0.0.1:35886)
0927 02:37:59.970827 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:59.970840 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:37:59.970937 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:37:59.971163 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:37:59.971166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:37:59.971379 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:37:59.971512 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:59.971520 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:59.972233 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:59.972237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:59.973107 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:37:59.973119 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:37:59.973542 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:37:59.973546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:37:59.973649 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:37:59.974077 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:37:59.974091 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:37:59.975311 (+ 1220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:37:59.976794 (+ 1483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:37:59.976797 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:37:59.976799 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:37:59.977033 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:37:59.977281 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:37:59.977284 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:37:59.977286 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:37:59.977362 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:37:59.977709 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:37:59.977712 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:37:59.977800 (+ 88us) client_negotiation.cc:769] Sending connection context
0927 02:37:59.977910 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 02:37:59.978040 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108}
I20260927 02:37:59.979161 22705 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:37:59.979765 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:59.979561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32819 (local address 127.0.0.1:50938)
0927 02:37:59.979664 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:59.979708 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 02:37:59.980334 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:37:59.980119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32819 (local address 127.0.0.1:50950)
0927 02:37:59.980243 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 02:37:59.980286 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260927 02:37:59.980449 22705 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32819: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32819: connect: Connection refused (error 111)
I20260927 02:38:00.011606 22705 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } attempt: 1
I20260927 02:38:00.012466 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:00.012136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32819 (local address 127.0.0.1:50952)
0927 02:38:00.012291 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:00.012371 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 02:38:00.013221 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:00.013005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32819 (local address 127.0.0.1:50958)
0927 02:38:00.013109 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:00.013160 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 02:38:00.013338 22705 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32819: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32819: connect: Connection refused (error 111)
I20260927 02:38:00.019652 22780 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } attempt: 1
I20260927 02:38:00.020319 22783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:00.020083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32819 (local address 127.0.0.1:50970)
0927 02:38:00.020201 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:00.020252 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 02:38:00.021032 22783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:00.020776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32819 (local address 127.0.0.1:50982)
0927 02:38:00.020873 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:00.020919 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 02:38:00.021142 22780 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32819: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32819: connect: Connection refused (error 111)
W20260927 02:38:00.072247 22786 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 02:38:00.072510 22786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:38:00.072543 22786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:38:00.072602 22786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:38:00.076488 22786 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:38:00.076541 22786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:38:00.076568 22786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:38:00.076588 22786 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:38:00.076615 22786 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:38:00.081634 22786 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44429,127.0.98.61:32887,127.0.98.60:32819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:32819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:38:00.082551 22786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:38:00.083444 22786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:38:00.088492 22705 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } attempt: 2
I20260927 02:38:00.089291 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:00.088976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32819 (local address 127.0.0.1:50986)
0927 02:38:00.089113 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:00.089180 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 02:38:00.089287 22780 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } attempt: 2
I20260927 02:38:00.089891 22783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:00.089654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32819 (local address 127.0.0.1:51002)
0927 02:38:00.089770 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:00.089817 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 02:38:00.090036 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:00.089813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32819 (local address 127.0.0.1:51016)
0927 02:38:00.089924 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:00.089973 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 02:38:00.090517 22705 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32819: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32819: connect: Connection refused (error 111)
I20260927 02:38:00.090826 22783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:00.090601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32819 (local address 127.0.0.1:51026)
0927 02:38:00.090689 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:00.090762 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260927 02:38:00.091024 22780 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32819: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32819: connect: Connection refused (error 111)
W20260927 02:38:00.091295 22793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:38:00.091353 22792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:38:00.091292 22795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:38:00.091631 22786 server_base.cc:1034] running on GCE node
I20260927 02:38:00.092110 22786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:38:00.093142 22786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:38:00.094298 22786 hybrid_clock.cc:648] HybridClock initialized: now 1790476680094284 us; error 28 us; skew 500 ppm
Sep 27 02:38:00 dist-test-slave-grlk krb5kdc[22624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:38:00.096864 22786 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 02:38:00.097730 22786 webserver.cc:460] Webserver started at http://127.0.98.60:42967/ using document root <none> and password file <none>
I20260927 02:38:00.098244 22786 fs_manager.cc:359] Metadata directory not provided
I20260927 02:38:00.098308 22786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:38:00.098496 22786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:38:00.100396 22786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/minicluster-data/master-2/data/instance:
uuid: "ff7f73ac95ef4efbbb46d51408200ea7"
format_stamp: "Formatted at 2026-09-27 02:38:00 on dist-test-slave-grlk"
server_key: "09899ac7be214377e263550170a260c7"
server_key_iv: "cf41f886bebba9c5cdba76ccc5d9313f"
server_key_version: "encryptionkey@0"
I20260927 02:38:00.100900 22786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/minicluster-data/master-2/wal/instance:
uuid: "ff7f73ac95ef4efbbb46d51408200ea7"
format_stamp: "Formatted at 2026-09-27 02:38:00 on dist-test-slave-grlk"
server_key: "09899ac7be214377e263550170a260c7"
server_key_iv: "cf41f886bebba9c5cdba76ccc5d9313f"
server_key_version: "encryptionkey@0"
I20260927 02:38:00.104298 22786 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 02:38:00.106701 22802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:00.107856 22786 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:38:00.107970 22786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/minicluster-data/master-2/wal
uuid: "ff7f73ac95ef4efbbb46d51408200ea7"
format_stamp: "Formatted at 2026-09-27 02:38:00 on dist-test-slave-grlk"
server_key: "09899ac7be214377e263550170a260c7"
server_key_iv: "cf41f886bebba9c5cdba76ccc5d9313f"
server_key_version: "encryptionkey@0"
I20260927 02:38:00.108084 22786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:38:00.122381 22786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:38:00.125324 22786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:38:00.125495 22786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:38:00.131951 22786 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:32819
I20260927 02:38:00.132202 22854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:32819 every 8 connection(s)
I20260927 02:38:00.132838 22786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/minicluster-data/master-2/data/info.pb
I20260927 02:38:00.135371 22855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:38:00.138792 22855 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:38:00.139880 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 22786
I20260927 02:38:00.139992 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476410933508-392-0/minicluster-data/master-2/wal/instance
I20260927 02:38:00.140268 392 external_mini_cluster.cc:1428] Setting key 23a3b0ed940b695dc8497f2b5a884aed
Sep 27 02:38:00 dist-test-slave-grlk krb5kdc[22624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 02:38:00 dist-test-slave-grlk krb5kdc[22624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 02:38:00.151152 22707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:00.140491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36138 (local address 127.0.98.62:44429)
0927 02:38:00.140624 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:00.140629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:00.141506 (+ 877us) server_negotiation.cc:407] Connection header received
0927 02:38:00.142337 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:00.142340 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:00.142390 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:00.142460 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:00.143304 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:00.143805 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:00.144818 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:00.145103 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:00.145179 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:00.147576 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:00.147580 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:00.147583 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:00.147609 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:00.149504 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:00.149904 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:00.149907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:00.149908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:00.149978 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:00.150248 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:00.150250 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:00.150251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:00.150414 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:00.150486 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:00.150856 (+ 370us) server_negotiation.cc:299] Negotiation successful
0927 02:38:00.150965 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 02:38:00.152016 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:00.140883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51040 (local address 127.0.98.60:32819)
0927 02:38:00.141404 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:00.141420 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:00.141454 (+ 34us) server_negotiation.cc:407] Connection header received
0927 02:38:00.142149 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:00.142165 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:00.142344 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:00.142495 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:00.143161 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:00.144081 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:00.144885 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:00.145337 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:00.145432 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:00.148050 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:00.148055 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:00.148064 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:00.148083 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:00.149465 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:00.149864 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:00.149870 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:00.149871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:00.149968 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:00.150205 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:00.150208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:00.150210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:00.150565 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:00.150743 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:00.150893 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 02:38:00.151081 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":414,"spinlock_wait_cycles":12160,"thread_start_us":147,"threads_started":1}
I20260927 02:38:00.152016 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:00.140883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44429 (local address 127.0.0.1:36138)
0927 02:38:00.141383 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:00.141419 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:00.142154 (+ 735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:00.142528 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:00.142542 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:00.142916 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:38:00.143162 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:00.143174 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:00.143902 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:00.143905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:00.144701 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:00.144709 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:00.145228 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:00.145232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:00.145382 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:00.145977 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:38:00.146000 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:38:00.147438 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:38:00.149605 (+ 2167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:00.149613 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:00.149622 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:00.149815 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:00.150062 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:00.150065 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:00.150067 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:00.150162 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:00.150506 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:38:00.150511 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:38:00.150631 (+ 120us) client_negotiation.cc:769] Sending connection context
0927 02:38:00.150790 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 02:38:00.151135 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":90,"thread_start_us":128,"threads_started":1}
I20260927 02:38:00.153129 22855 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32887 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:38:00 dist-test-slave-grlk krb5kdc[22624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 02:38:00 dist-test-slave-grlk krb5kdc[22624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:38:00.161480 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:00.153579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32887 (local address 127.0.0.1:35894)
0927 02:38:00.154020 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:00.154032 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:00.154096 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:00.154402 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:00.154404 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:00.154579 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:38:00.154686 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:00.154692 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:00.155373 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:00.155377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:00.155898 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:00.155905 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:00.156307 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:00.156311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:00.156400 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:00.156894 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:38:00.156909 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:38:00.158412 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:38:00.160136 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:00.160138 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:00.160140 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:00.160310 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:00.160566 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:00.160568 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:00.160569 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:00.160634 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:00.161026 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:38:00.161038 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:38:00.161132 (+ 94us) client_negotiation.cc:769] Sending connection context
0927 02:38:00.161211 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 02:38:00.161322 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385}
I20260927 02:38:00.161495 22785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:00.153723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35894 (local address 127.0.98.61:32887)
0927 02:38:00.153847 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:00.153850 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:00.154087 (+ 237us) server_negotiation.cc:407] Connection header received
0927 02:38:00.154174 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:00.154177 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:00.154217 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:00.154286 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:00.154794 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:00.155279 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:00.156000 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:00.156216 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:00.156299 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:00.158545 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:00.158548 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:00.158550 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:00.158573 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:00.160035 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:00.160400 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:00.160403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:00.160405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:00.160487 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:00.160711 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:00.160714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:00.160716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:00.160922 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:00.161033 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:00.161283 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 02:38:00.161384 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 02:38:00.162178 22855 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:38:00.162307 22707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:00.154330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36152 (local address 127.0.98.62:44429)
0927 02:38:00.154436 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:00.154440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:00.154451 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:38:00.154486 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:00.154489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:00.154526 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:00.154603 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:00.155222 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:00.155697 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:00.156402 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:00.156725 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:00.156815 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:00.159221 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:00.159225 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:00.159228 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:00.159253 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:00.160529 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:00.160954 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:00.160958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:00.160960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:00.161044 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:00.161298 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:00.161301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:00.161303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:00.161511 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:00.161744 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:00.162057 (+ 313us) server_negotiation.cc:299] Negotiation successful
0927 02:38:00.162175 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 02:38:00 dist-test-slave-grlk krb5kdc[22624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 02:38:00 dist-test-slave-grlk krb5kdc[22624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 02:38:00.169013 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:00.162711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51046 (local address 127.0.98.60:32819)
0927 02:38:00.162804 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:00.162808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:00.162843 (+ 35us) server_negotiation.cc:407] Connection header received
0927 02:38:00.162874 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:00.162877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:00.162917 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:00.162979 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:00.163295 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:00.163936 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:00.164599 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:00.164826 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:00.164902 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:00.166508 (+ 1606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:00.166511 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:00.166514 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:00.166534 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:00.167663 (+ 1129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:00.168007 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:00.168010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:00.168011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:00.168088 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:00.168322 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:00.168325 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:00.168327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:00.168467 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:00.168542 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:00.168745 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 02:38:00.168862 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":73,"server-negotiator.queue_time_us":34}
I20260927 02:38:00.169013 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:00.162609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32819 (local address 127.0.0.1:51046)
0927 02:38:00.162709 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:00.162722 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:00.162789 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:00.162989 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:00.162991 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:00.163111 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:38:00.163195 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:00.163202 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:00.164023 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:00.164026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:00.164508 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:00.164514 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:00.164908 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:00.164912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:00.165038 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:00.165407 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:38:00.165416 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:38:00.166403 (+ 987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:38:00.167758 (+ 1355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:00.167760 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:00.167761 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:00.167926 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:00.168174 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:00.168177 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:00.168179 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:00.168250 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:00.168544 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:38:00.168547 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:38:00.168619 (+ 72us) client_negotiation.cc:769] Sending connection context
0927 02:38:00.168697 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 02:38:00.168812 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":18}
I20260927 02:38:00.171351 22785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:00.163346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35900 (local address 127.0.98.61:32887)
0927 02:38:00.163439 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:00.163443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:00.163477 (+ 34us) server_negotiation.cc:407] Connection header received
0927 02:38:00.163561 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:00.163563 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:00.163600 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:00.163672 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:00.164169 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:00.164630 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:00.165474 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:00.165693 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:00.165783 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:00.168437 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:00.168440 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:00.168443 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:00.168466 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:00.169902 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:00.170259 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:00.170262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:00.170263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:00.170337 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:00.170535 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:00.170537 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:00.170539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:00.170716 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:00.170931 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:00.171003 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 02:38:00.171220 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 02:38:00.172104 22855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff7f73ac95ef4efbbb46d51408200ea7: Bootstrap starting.
I20260927 02:38:00.174422 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:38:00.174896 22855 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff7f73ac95ef4efbbb46d51408200ea7: Neither blocks nor log segments found. Creating new log.
I20260927 02:38:00.175589 22855 log.cc:824] T 00000000000000000000000000000000 P ff7f73ac95ef4efbbb46d51408200ea7: Log is configured to *not* fsync() on all Append() calls
I20260927 02:38:00.175617 22679 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "c7836ed54a054c61acf1fa501075cf72"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36152
I20260927 02:38:00.177625 22855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff7f73ac95ef4efbbb46d51408200ea7: No bootstrap required, opened a new log
I20260927 02:38:00.181388 22855 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff7f73ac95ef4efbbb46d51408200ea7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7836ed54a054c61acf1fa501075cf72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "91118d43f3224924902d724c521d7a99" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32887 } } peers { permanent_uuid: "ff7f73ac95ef4efbbb46d51408200ea7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } }
I20260927 02:38:00.181632 22855 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff7f73ac95ef4efbbb46d51408200ea7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff7f73ac95ef4efbbb46d51408200ea7, State: Initialized, Role: FOLLOWER
I20260927 02:38:00.181945 22855 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff7f73ac95ef4efbbb46d51408200ea7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7836ed54a054c61acf1fa501075cf72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "91118d43f3224924902d724c521d7a99" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32887 } } peers { permanent_uuid: "ff7f73ac95ef4efbbb46d51408200ea7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } }
I20260927 02:38:00.182127 22780 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } attempt: 3
I20260927 02:38:00.182428 22861 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff7f73ac95ef4efbbb46d51408200ea7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7836ed54a054c61acf1fa501075cf72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "91118d43f3224924902d724c521d7a99" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32887 } } peers { permanent_uuid: "ff7f73ac95ef4efbbb46d51408200ea7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } } }
I20260927 02:38:00.182749 22861 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff7f73ac95ef4efbbb46d51408200ea7 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:38:00.183243 22855 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff7f73ac95ef4efbbb46d51408200ea7 [sys.catalog]: configured and running, proceeding with master startup.
Sep 27 02:38:00 dist-test-slave-grlk krb5kdc[22624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
W20260927 02:38:00.187901 22872 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ff7f73ac95ef4efbbb46d51408200ea7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:38:00.187974 22872 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:38:00.189682 22705 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } attempt: 3
I20260927 02:38:00.187458 22786 master_runner.cc:186] Error getting master registration for 127.0.98.62:44429: OK, instance_id { permanent_uuid: "c7836ed54a054c61acf1fa501075cf72" instance_seqno: 1790476679731286 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 02:38:00.191223 22786 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 27 02:38:00 dist-test-slave-grlk krb5kdc[22624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 02:38:00.194900 22783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:00.182455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32819 (local address 127.0.0.1:51048)
0927 02:38:00.182580 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:00.182598 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:00.182668 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:00.182971 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:00.182973 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:00.183127 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:38:00.183202 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:00.183207 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:00.183881 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:00.183884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:00.184528 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:00.184537 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:00.185051 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:00.185055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:00.185143 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:00.185604 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:38:00.185618 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:38:00.191593 (+ 5975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:38:00.193398 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:00.193401 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:00.193403 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:00.193595 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:00.193847 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:00.193850 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:00.193852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:00.193940 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:00.194386 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:38:00.194389 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:38:00.194491 (+ 102us) client_negotiation.cc:769] Sending connection context
0927 02:38:00.194604 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 02:38:00.194735 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 02:38:00.194939 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:00.182677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51048 (local address 127.0.98.60:32819)
0927 02:38:00.182785 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:00.182789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:00.182800 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:38:00.182831 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:00.182833 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:00.182872 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:00.182934 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:00.183303 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:00.183782 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:00.184627 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:00.184919 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:00.185045 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:00.191717 (+ 6672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:00.191721 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:00.191723 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:00.191746 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:00.193297 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:00.193693 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:00.193696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:00.193698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:00.193770 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:00.194039 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:00.194043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:00.194045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:00.194288 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:00.194394 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:00.194672 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 02:38:00.194785 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 02:38:00.197554 22780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91118d43f3224924902d724c521d7a99: Bootstrap starting.
I20260927 02:38:00.199615 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:00.190044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32819 (local address 127.0.0.1:51052)
0927 02:38:00.190202 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:00.190214 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:00.190282 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:00.190671 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:00.190675 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:00.190833 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:38:00.190914 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:00.190919 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:00.191935 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:00.191939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:00.192736 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:00.192748 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:00.193362 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:00.193366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:00.193456 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:00.193898 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:38:00.193911 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:38:00.195635 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:38:00.198149 (+ 2514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:00.198153 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:00.198155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:00.198380 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:00.198675 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:00.198678 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:00.198680 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:00.198768 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:00.199146 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:38:00.199149 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:38:00.199242 (+ 93us) client_negotiation.cc:769] Sending connection context
0927 02:38:00.199344 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 02:38:00.199465 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260927 02:38:00.199710 22874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:00.190126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51052 (local address 127.0.98.60:32819)
0927 02:38:00.190456 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:00.190465 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:00.190477 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:38:00.190505 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:00.190508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:00.190568 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:00.190641 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:00.191015 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:00.191822 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:00.192866 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:00.193248 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:00.193347 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:00.195755 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:00.195759 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:00.195761 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:00.195785 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:00.198040 (+ 2255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:00.198474 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:00.198477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:00.198479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:00.198582 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:00.198857 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:00.198860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:00.198862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:00.199053 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:00.199145 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:00.199404 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 02:38:00.199516 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":109,"threads_started":1}
I20260927 02:38:00.200017 22780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91118d43f3224924902d724c521d7a99: Neither blocks nor log segments found. Creating new log.
I20260927 02:38:00.200737 22780 log.cc:824] T 00000000000000000000000000000000 P 91118d43f3224924902d724c521d7a99: Log is configured to *not* fsync() on all Append() calls
I20260927 02:38:00.202754 22780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91118d43f3224924902d724c521d7a99: No bootstrap required, opened a new log
I20260927 02:38:00.202795 22705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7836ed54a054c61acf1fa501075cf72: Bootstrap starting.
I20260927 02:38:00.205418 22705 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7836ed54a054c61acf1fa501075cf72: Neither blocks nor log segments found. Creating new log.
I20260927 02:38:00.206221 22705 log.cc:824] T 00000000000000000000000000000000 P c7836ed54a054c61acf1fa501075cf72: Log is configured to *not* fsync() on all Append() calls
I20260927 02:38:00.206614 22780 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91118d43f3224924902d724c521d7a99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7836ed54a054c61acf1fa501075cf72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "91118d43f3224924902d724c521d7a99" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32887 } } peers { permanent_uuid: "ff7f73ac95ef4efbbb46d51408200ea7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } }
I20260927 02:38:00.206836 22780 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91118d43f3224924902d724c521d7a99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91118d43f3224924902d724c521d7a99, State: Initialized, Role: FOLLOWER
I20260927 02:38:00.207162 22780 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91118d43f3224924902d724c521d7a99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7836ed54a054c61acf1fa501075cf72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "91118d43f3224924902d724c521d7a99" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32887 } } peers { permanent_uuid: "ff7f73ac95ef4efbbb46d51408200ea7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } }
I20260927 02:38:00.207653 22877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91118d43f3224924902d724c521d7a99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7836ed54a054c61acf1fa501075cf72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "91118d43f3224924902d724c521d7a99" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32887 } } peers { permanent_uuid: "ff7f73ac95ef4efbbb46d51408200ea7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } } }
I20260927 02:38:00.207849 22877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91118d43f3224924902d724c521d7a99 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:38:00.208353 22705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7836ed54a054c61acf1fa501075cf72: No bootstrap required, opened a new log
I20260927 02:38:00.208487 22780 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91118d43f3224924902d724c521d7a99 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:38:00.211429 22705 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7836ed54a054c61acf1fa501075cf72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7836ed54a054c61acf1fa501075cf72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "91118d43f3224924902d724c521d7a99" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32887 } } peers { permanent_uuid: "ff7f73ac95ef4efbbb46d51408200ea7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } }
I20260927 02:38:00.211724 22705 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7836ed54a054c61acf1fa501075cf72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7836ed54a054c61acf1fa501075cf72, State: Initialized, Role: FOLLOWER
I20260927 02:38:00.212167 22705 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7836ed54a054c61acf1fa501075cf72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7836ed54a054c61acf1fa501075cf72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "91118d43f3224924902d724c521d7a99" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32887 } } peers { permanent_uuid: "ff7f73ac95ef4efbbb46d51408200ea7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } }
I20260927 02:38:00.213581 22705 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7836ed54a054c61acf1fa501075cf72 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:38:00.213649 22679 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7836ed54a054c61acf1fa501075cf72 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 02:38:00.213779 22679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7836ed54a054c61acf1fa501075cf72 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:38:00.214942 22679 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7836ed54a054c61acf1fa501075cf72 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7836ed54a054c61acf1fa501075cf72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "91118d43f3224924902d724c521d7a99" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32887 } } peers { permanent_uuid: "ff7f73ac95ef4efbbb46d51408200ea7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } }
I20260927 02:38:00.211710 22712 master_runner.cc:186] Error getting master registration for 127.0.98.62:44429: OK, instance_id { permanent_uuid: "c7836ed54a054c61acf1fa501075cf72" instance_seqno: 1790476679731286 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 02:38:00.215342 22712 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 02:38:00.216298 22754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c7836ed54a054c61acf1fa501075cf72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "91118d43f3224924902d724c521d7a99"
I20260927 02:38:00.216502 22754 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91118d43f3224924902d724c521d7a99 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:38:00.216446 22829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c7836ed54a054c61acf1fa501075cf72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ff7f73ac95ef4efbbb46d51408200ea7"
I20260927 02:38:00.216660 22829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff7f73ac95ef4efbbb46d51408200ea7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:38:00.217692 22754 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 91118d43f3224924902d724c521d7a99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7836ed54a054c61acf1fa501075cf72 in term 1.
I20260927 02:38:00.217844 22829 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ff7f73ac95ef4efbbb46d51408200ea7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7836ed54a054c61acf1fa501075cf72 in term 1.
I20260927 02:38:00.218092 22656 leader_election.cc:302] T 00000000000000000000000000000000 P c7836ed54a054c61acf1fa501075cf72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91118d43f3224924902d724c521d7a99, c7836ed54a054c61acf1fa501075cf72; no voters:
I20260927 02:38:00.215854 22679 leader_election.cc:288] T 00000000000000000000000000000000 P c7836ed54a054c61acf1fa501075cf72 [CANDIDATE]: Term 1 election: Requested vote from peers 91118d43f3224924902d724c521d7a99 (127.0.98.61:32887), ff7f73ac95ef4efbbb46d51408200ea7 (127.0.98.60:32819)
W20260927 02:38:00.218843 22901 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c7836ed54a054c61acf1fa501075cf72: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:38:00.218914 22901 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260927 02:38:00.221341 22888 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 91118d43f3224924902d724c521d7a99: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:38:00.221408 22888 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:38:00.225217 22903 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7836ed54a054c61acf1fa501075cf72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7836ed54a054c61acf1fa501075cf72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "91118d43f3224924902d724c521d7a99" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32887 } } peers { permanent_uuid: "ff7f73ac95ef4efbbb46d51408200ea7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } } }
I20260927 02:38:00.225411 22903 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7836ed54a054c61acf1fa501075cf72 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:38:00.225523 22903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7836ed54a054c61acf1fa501075cf72 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:38:00.225615 22903 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7836ed54a054c61acf1fa501075cf72 [term 1 LEADER]: Becoming Leader. State: Replica: c7836ed54a054c61acf1fa501075cf72, State: Running, Role: LEADER
I20260927 02:38:00.226086 22903 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7836ed54a054c61acf1fa501075cf72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7836ed54a054c61acf1fa501075cf72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "91118d43f3224924902d724c521d7a99" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32887 } } peers { permanent_uuid: "ff7f73ac95ef4efbbb46d51408200ea7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } }
I20260927 02:38:00.229063 22903 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7836ed54a054c61acf1fa501075cf72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7836ed54a054c61acf1fa501075cf72. Latest consensus state: current_term: 1 leader_uuid: "c7836ed54a054c61acf1fa501075cf72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7836ed54a054c61acf1fa501075cf72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "91118d43f3224924902d724c521d7a99" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32887 } } peers { permanent_uuid: "ff7f73ac95ef4efbbb46d51408200ea7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } } }
I20260927 02:38:00.229568 22903 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7836ed54a054c61acf1fa501075cf72 [sys.catalog]: This master's current role is: LEADER
I20260927 02:38:00.231353 22874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:00.220291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51058 (local address 127.0.98.60:32819)
0927 02:38:00.220392 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:00.220397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:00.221713 (+ 1316us) server_negotiation.cc:407] Connection header received
0927 02:38:00.221821 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:00.221824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:00.221900 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:00.221998 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:00.223021 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:00.224116 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:00.225341 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:00.225678 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:00.225764 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:00.227739 (+ 1975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:00.227743 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:00.227745 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:00.227773 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:00.229445 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:00.230045 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:00.230049 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:00.230051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:00.230136 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:00.230415 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:00.230418 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:00.230419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:00.230635 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:00.230734 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:00.231071 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 02:38:00.231192 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 02:38:00.232884 22911 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260927 02:38:00.233069 22658 tablet.cc:2369] T 00000000000000000000000000000000 P c7836ed54a054c61acf1fa501075cf72: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:38:00.233683 22637 master_runner.cc:263] No leader master found from master c7836ed54a054c61acf1fa501075cf72
I20260927 02:38:00.233742 22637 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 02:38:00.233875 22911 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:38:00.239141 22707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:00.222081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36154 (local address 127.0.98.62:44429)
0927 02:38:00.222182 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:00.222186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:00.227261 (+ 5075us) server_negotiation.cc:407] Connection header received
0927 02:38:00.227432 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:00.227435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:00.227485 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:00.227535 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:00.229043 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:00.229808 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:00.232887 (+ 3079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:00.233221 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:00.233293 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:00.235816 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:00.235819 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:00.235822 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:00.235846 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:00.237406 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:00.237876 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:00.237880 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:00.237882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:00.237959 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:00.238228 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:00.238232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:00.238233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:00.238429 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:00.238522 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:00.238883 (+ 361us) server_negotiation.cc:299] Negotiation successful
0927 02:38:00.238994 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 02:38:00.244529 22829 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ff7f73ac95ef4efbbb46d51408200ea7 [term 1 FOLLOWER]: Refusing update from remote peer c7836ed54a054c61acf1fa501075cf72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:38:00.245241 22903 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c7836ed54a054c61acf1fa501075cf72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff7f73ac95ef4efbbb46d51408200ea7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:38:00.248950 22861 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff7f73ac95ef4efbbb46d51408200ea7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7836ed54a054c61acf1fa501075cf72. Latest consensus state: current_term: 1 leader_uuid: "c7836ed54a054c61acf1fa501075cf72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7836ed54a054c61acf1fa501075cf72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "91118d43f3224924902d724c521d7a99" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32887 } } peers { permanent_uuid: "ff7f73ac95ef4efbbb46d51408200ea7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } } }
I20260927 02:38:00.249084 22861 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff7f73ac95ef4efbbb46d51408200ea7 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:38:00.251120 22754 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 91118d43f3224924902d724c521d7a99 [term 1 FOLLOWER]: Refusing update from remote peer c7836ed54a054c61acf1fa501075cf72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:38:00.251693 22903 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c7836ed54a054c61acf1fa501075cf72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91118d43f3224924902d724c521d7a99" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:38:00.255117 22877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91118d43f3224924902d724c521d7a99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7836ed54a054c61acf1fa501075cf72. Latest consensus state: current_term: 1 leader_uuid: "c7836ed54a054c61acf1fa501075cf72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7836ed54a054c61acf1fa501075cf72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "91118d43f3224924902d724c521d7a99" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32887 } } peers { permanent_uuid: "ff7f73ac95ef4efbbb46d51408200ea7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } } }
I20260927 02:38:00.255242 22877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91118d43f3224924902d724c521d7a99 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:38:00.260329 22903 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7836ed54a054c61acf1fa501075cf72 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c7836ed54a054c61acf1fa501075cf72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7836ed54a054c61acf1fa501075cf72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "91118d43f3224924902d724c521d7a99" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32887 } } peers { permanent_uuid: "ff7f73ac95ef4efbbb46d51408200ea7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } } }
I20260927 02:38:00.260447 22903 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7836ed54a054c61acf1fa501075cf72 [sys.catalog]: This master's current role is: LEADER
I20260927 02:38:00.261458 22877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91118d43f3224924902d724c521d7a99 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c7836ed54a054c61acf1fa501075cf72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7836ed54a054c61acf1fa501075cf72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "91118d43f3224924902d724c521d7a99" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32887 } } peers { permanent_uuid: "ff7f73ac95ef4efbbb46d51408200ea7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } } }
I20260927 02:38:00.261567 22877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91118d43f3224924902d724c521d7a99 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:38:00.262080 22912 mvcc.cc:205] Tried to move back new op lower bound from 7333792482273947648 to 7333792482211614720. Current Snapshot: MvccSnapshot[applied={T|T < 7333792482273947648}]
I20260927 02:38:00.263999 22915 mvcc.cc:205] Tried to move back new op lower bound from 7333792482273947648 to 7333792482211614720. Current Snapshot: MvccSnapshot[applied={T|T < 7333792482273947648}]
I20260927 02:38:00.264704 22890 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7836ed54a054c61acf1fa501075cf72 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c7836ed54a054c61acf1fa501075cf72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7836ed54a054c61acf1fa501075cf72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "91118d43f3224924902d724c521d7a99" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32887 } } peers { permanent_uuid: "ff7f73ac95ef4efbbb46d51408200ea7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } } }
I20260927 02:38:00.264853 22890 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7836ed54a054c61acf1fa501075cf72 [sys.catalog]: This master's current role is: LEADER
I20260927 02:38:00.265889 22861 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff7f73ac95ef4efbbb46d51408200ea7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c7836ed54a054c61acf1fa501075cf72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7836ed54a054c61acf1fa501075cf72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44429 } } peers { permanent_uuid: "91118d43f3224924902d724c521d7a99" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32887 } } peers { permanent_uuid: "ff7f73ac95ef4efbbb46d51408200ea7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32819 } } }
I20260927 02:38:00.266012 22861 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff7f73ac95ef4efbbb46d51408200ea7 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:38:00.269162 22913 mvcc.cc:205] Tried to move back new op lower bound from 7333792482273947648 to 7333792482211614720. Current Snapshot: MvccSnapshot[applied={T|T < 7333792482273947648}]
I20260927 02:38:00.270862 22911 catalog_manager.cc:1347] Generated new cluster ID: a5851d3b82c84f57910eddbcde9e57c0
I20260927 02:38:00.270925 22911 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:38:00.293812 22911 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:38:00.295001 22911 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:38:00.310168 22911 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7836ed54a054c61acf1fa501075cf72: Generated new TSK 0
I20260927 02:38:00.311138 22911 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:38:00.320163 22785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:00.253291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35906 (local address 127.0.98.61:32887)
0927 02:38:00.253400 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:00.253404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:00.253818 (+ 414us) server_negotiation.cc:407] Connection header received
0927 02:38:00.253857 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:00.253860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:00.253904 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:00.253963 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:00.254810 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:00.255590 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:00.256676 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:00.257032 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:00.257120 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:00.274157 (+ 17037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:00.274161 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:00.274164 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:00.274192 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:00.317751 (+ 43559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:00.318344 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:00.318348 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:00.318350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:00.318447 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:00.318768 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:00.318772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:00.318773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:00.319001 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:00.319132 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:00.319886 (+ 754us) server_negotiation.cc:299] Negotiation successful
0927 02:38:00.320008 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 02:38:00.375435 22707 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:38:00.373795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36164 (local address 127.0.98.62:44429)
0927 02:38:00.373899 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:00.373904 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:00.374517 (+ 613us) server_negotiation.cc:407] Connection header received
0927 02:38:00.374671 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:00.374674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:00.374722 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:00.374793 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:00.375295 (+ 502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36164: BlockingRecv error: recv got EOF from 127.0.0.1:36164 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260927 02:38:00.376205 22874 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:38:00.373384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51060 (local address 127.0.98.60:32819)
0927 02:38:00.373537 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:00.373542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:00.375570 (+ 2028us) server_negotiation.cc:407] Connection header received
0927 02:38:00.375614 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:00.375617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:00.375660 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:00.375740 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:00.376094 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51060: BlockingRecv error: recv got EOF from 127.0.0.1:51060 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260927 02:38:00.376979 22785 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:38:00.374040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35920 (local address 127.0.98.61:32887)
0927 02:38:00.374158 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:00.374163 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:00.376139 (+ 1976us) server_negotiation.cc:407] Connection header received
0927 02:38:00.376303 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:00.376307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:00.376356 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:00.376407 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:00.376851 (+ 444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35920: BlockingRecv error: recv got EOF from 127.0.0.1:35920 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260927 02:38:00.386755 22785 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:38:00.385261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35930 (local address 127.0.98.61:32887)
0927 02:38:00.385364 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:00.385368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:00.385786 (+ 418us) server_negotiation.cc:407] Connection header received
0927 02:38:00.385997 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:00.386002 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:00.386089 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:00.386139 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:00.386649 (+ 510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35930: BlockingRecv error: recv got EOF from 127.0.0.1:35930 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260927 02:38:00.391724 22785 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:38:00.390229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35938 (local address 127.0.98.61:32887)
0927 02:38:00.390326 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:00.390330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:00.390837 (+ 507us) server_negotiation.cc:407] Connection header received
0927 02:38:00.391023 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:00.391027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:00.391082 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:00.391132 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:00.391620 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35938: BlockingRecv error: recv got EOF from 127.0.0.1:35938 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260927 02:38:00.396452 22785 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:38:00.395524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35954 (local address 127.0.98.61:32887)
0927 02:38:00.395620 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:00.395624 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:00.395819 (+ 195us) server_negotiation.cc:407] Connection header received
0927 02:38:00.395908 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:00.395911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:00.395951 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:00.396022 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:00.396365 (+ 343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35954: BlockingRecv error: recv got EOF from 127.0.0.1:35954 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 02:38:00.402798 22785 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:38:00.401820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35962 (local address 127.0.98.61:32887)
0927 02:38:00.401956 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:00.401960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:00.402126 (+ 166us) server_negotiation.cc:407] Connection header received
0927 02:38:00.402228 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:00.402231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:00.402273 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:00.402334 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:00.402701 (+ 367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35962: BlockingRecv error: recv got EOF from 127.0.0.1:35962 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260927 02:38:00.413616 22785 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:38:00.412470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35974 (local address 127.0.98.61:32887)
0927 02:38:00.412571 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:00.412574 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:00.412769 (+ 195us) server_negotiation.cc:407] Connection header received
0927 02:38:00.412922 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:00.412925 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:00.412994 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:00.413064 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:00.413521 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35974: BlockingRecv error: recv got EOF from 127.0.0.1:35974 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 02:38:00.432453 22785 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:38:00.431313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35980 (local address 127.0.98.61:32887)
0927 02:38:00.431432 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:00.431437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:00.431668 (+ 231us) server_negotiation.cc:407] Connection header received
0927 02:38:00.431798 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:00.431801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:00.431847 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:00.431912 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:00.432332 (+ 420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35980: BlockingRecv error: recv got EOF from 127.0.0.1:35980 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 02:38:00.467224 22785 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:38:00.466249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35994 (local address 127.0.98.61:32887)
0927 02:38:00.466387 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:00.466391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:00.466463 (+ 72us) server_negotiation.cc:407] Connection header received
0927 02:38:00.466578 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:00.466581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:00.466634 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:00.466732 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:00.467108 (+ 376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35994: BlockingRecv error: recv got EOF from 127.0.0.1:35994 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 02:38:00.534129 22785 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:38:00.533180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36006 (local address 127.0.98.61:32887)
0927 02:38:00.533334 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:00.533337 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:00.533443 (+ 106us) server_negotiation.cc:407] Connection header received
0927 02:38:00.533523 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:00.533526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:00.533576 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:00.533681 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:00.534030 (+ 349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36006: BlockingRecv error: recv got EOF from 127.0.0.1:36006 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260927 02:38:00.665264 22785 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:38:00.664154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37528 (local address 127.0.98.61:32887)
0927 02:38:00.664310 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:00.664314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:00.664423 (+ 109us) server_negotiation.cc:407] Connection header received
0927 02:38:00.664567 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:00.664570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:00.664619 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:00.664688 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:00.665144 (+ 456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37528: BlockingRecv error: recv got EOF from 127.0.0.1:37528 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260927 02:38:00.924399 22785 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:38:00.923226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37530 (local address 127.0.98.61:32887)
0927 02:38:00.923371 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:00.923375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:00.923514 (+ 139us) server_negotiation.cc:407] Connection header received
0927 02:38:00.923700 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:00.923703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:00.923751 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:00.923819 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:00.924280 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37530: BlockingRecv error: recv got EOF from 127.0.0.1:37530 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 02:38:01.189205 22872 catalog_manager.cc:1259] Loaded cluster ID: a5851d3b82c84f57910eddbcde9e57c0
I20260927 02:38:01.189280 22872 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ff7f73ac95ef4efbbb46d51408200ea7: loading cluster ID for follower catalog manager: success
I20260927 02:38:01.190848 22872 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ff7f73ac95ef4efbbb46d51408200ea7: acquiring CA information for follower catalog manager: success
I20260927 02:38:01.191892 22872 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ff7f73ac95ef4efbbb46d51408200ea7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 02:38:01.222579 22888 catalog_manager.cc:1259] Loaded cluster ID: a5851d3b82c84f57910eddbcde9e57c0
I20260927 02:38:01.222661 22888 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 91118d43f3224924902d724c521d7a99: loading cluster ID for follower catalog manager: success
I20260927 02:38:01.224709 22888 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 91118d43f3224924902d724c521d7a99: acquiring CA information for follower catalog manager: success
I20260927 02:38:01.226159 22888 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 91118d43f3224924902d724c521d7a99: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 02:38:01.442026 22984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:01.438562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37540 (local address 127.0.98.61:32887)
0927 02:38:01.438768 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:01.438772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:01.438790 (+ 18us) server_negotiation.cc:407] Connection header received
0927 02:38:01.438943 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:01.438946 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:01.439077 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:01.439195 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:38:01.439403 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:01.439914 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:01.440696 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:01.440904 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:01.440994 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:01.441298 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:38:01.441709 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:38:01.441798 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:01.441829 (+ 31us) server_negotiation.cc:299] Negotiation successful
0927 02:38:01.441872 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":59,"threads_started":1}
I20260927 02:38:01.447710 22990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:01.444708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33082 (local address 127.0.98.60:32819)
0927 02:38:01.444975 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:01.444979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:01.444991 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:38:01.445059 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:01.445061 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:01.445165 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:01.445246 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:38:01.445393 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:01.445880 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:01.446590 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:01.446788 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:01.446850 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:01.447039 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:38:01.447374 (+ 335us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:38:01.447455 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:01.447523 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 02:38:01.447566 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":75,"threads_started":1}
I20260927 02:38:01.453720 22990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:01.450423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33098 (local address 127.0.98.60:32819)
0927 02:38:01.450582 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:01.450586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:01.450720 (+ 134us) server_negotiation.cc:407] Connection header received
0927 02:38:01.450816 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:01.450818 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:01.450851 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:01.450908 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:38:01.451173 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:01.451792 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:01.452631 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:01.452971 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:01.453068 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:01.453277 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:38:01.453366 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:38:01.453439 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:01.453533 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 02:38:01.453602 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 02:38:01.454434 22984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:01.450249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37550 (local address 127.0.98.61:32887)
0927 02:38:01.450351 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:01.450355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:01.451348 (+ 993us) server_negotiation.cc:407] Connection header received
0927 02:38:01.451444 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:01.451447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:01.451493 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:01.451555 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:38:01.451761 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:01.452411 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:01.453347 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:01.453648 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:01.453745 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:01.454053 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:38:01.454126 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:38:01.454192 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:01.454274 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 02:38:01.454319 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 02:38:01.454561 22996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:01.450250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38448 (local address 127.0.98.62:44429)
0927 02:38:01.450472 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:01.450476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:01.450493 (+ 17us) server_negotiation.cc:407] Connection header received
0927 02:38:01.450688 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:01.450692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:01.450833 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:01.450932 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:38:01.451269 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:01.451994 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:01.453112 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:01.453394 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:01.453463 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:01.453861 (+ 398us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:38:01.454136 (+ 275us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:38:01.454211 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:01.454345 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 02:38:01.454404 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":67,"threads_started":1}
I20260927 02:38:01.457477 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 22637
I20260927 02:38:01.462879 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 22712
I20260927 02:38:01.467816 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 22786
2026-09-27T02:38:01Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2967 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:38:01 dist-test-slave-grlk krb5kdc[23001](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:38:01 dist-test-slave-grlk krb5kdc[23001](info): set up 2 sockets
Sep 27 02:38:01 dist-test-slave-grlk krb5kdc[23001](info): commencing operation
krb5kdc: starting...
W20260927 02:38:02.424921 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/krb5kdc/test-user.keytab.
Sep 27 02:38:02 dist-test-slave-grlk krb5kdc[23001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:38:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:38:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:38:02.541657 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42593
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42593,127.0.98.61:34259,127.0.98.60:36527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:38:02.649106 23014 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 02:38:02.649382 23014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:38:02.649423 23014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:38:02.649485 23014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:38:02.653399 23014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:38:02.653445 23014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:38:02.653471 23014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:38:02.653512 23014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:38:02.653540 23014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:38:02.658567 23014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42593,127.0.98.61:34259,127.0.98.60:36527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:38:02.659487 23014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:38:02.660408 23014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:38:02.668267 23020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:38:02.668304 23014 server_base.cc:1034] running on GCE node
W20260927 02:38:02.668226 23022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:38:02.668226 23019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:38:02.669129 23014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:38:02.670128 23014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:38:02.671308 23014 hybrid_clock.cc:648] HybridClock initialized: now 1790476682671291 us; error 33 us; skew 500 ppm
Sep 27 02:38:02 dist-test-slave-grlk krb5kdc[23001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:38:02.678128 23014 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:38:02.679065 23014 webserver.cc:460] Webserver started at http://127.0.98.62:44131/ using document root <none> and password file <none>
I20260927 02:38:02.679580 23014 fs_manager.cc:359] Metadata directory not provided
I20260927 02:38:02.679649 23014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:38:02.679814 23014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:38:02.681813 23014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/minicluster-data/master-0/data/instance:
uuid: "475e5e78e82f4710a3cb95975c35d7e6"
format_stamp: "Formatted at 2026-09-27 02:38:02 on dist-test-slave-grlk"
server_key: "5223d7e17de4e19181b8f48cabd6640f"
server_key_iv: "ef859abe16305db4c13dcacb519acba8"
server_key_version: "encryptionkey@0"
I20260927 02:38:02.682297 23014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/minicluster-data/master-0/wal/instance:
uuid: "475e5e78e82f4710a3cb95975c35d7e6"
format_stamp: "Formatted at 2026-09-27 02:38:02 on dist-test-slave-grlk"
server_key: "5223d7e17de4e19181b8f48cabd6640f"
server_key_iv: "ef859abe16305db4c13dcacb519acba8"
server_key_version: "encryptionkey@0"
I20260927 02:38:02.685775 23014 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:38:02.688089 23029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:02.689436 23014 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:38:02.689556 23014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/minicluster-data/master-0/wal
uuid: "475e5e78e82f4710a3cb95975c35d7e6"
format_stamp: "Formatted at 2026-09-27 02:38:02 on dist-test-slave-grlk"
server_key: "5223d7e17de4e19181b8f48cabd6640f"
server_key_iv: "ef859abe16305db4c13dcacb519acba8"
server_key_version: "encryptionkey@0"
I20260927 02:38:02.689651 23014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:38:02.722560 23014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:38:02.725342 23014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:38:02.725505 23014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:38:02.732609 23014 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42593
I20260927 02:38:02.732841 23081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42593 every 8 connection(s)
I20260927 02:38:02.733633 23014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:38:02.736193 23082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:38:02.737635 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 23014
I20260927 02:38:02.737767 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/minicluster-data/master-0/wal/instance
I20260927 02:38:02.738029 392 external_mini_cluster.cc:1428] Setting key 7809fdcb57cecbbbab92dea681fc4e25
I20260927 02:38:02.739751 23082 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42593 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 27 02:38:02 dist-test-slave-grlk krb5kdc[23001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:38:02.754598 23085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:02.741766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42593 (local address 127.0.0.1:60240)
0927 02:38:02.742203 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:02.742252 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:02.743041 (+ 789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:02.743807 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:02.743811 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:02.744236 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:38:02.744598 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:02.744618 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:02.746205 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:02.746208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:02.746986 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:02.746995 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:02.747562 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:02.747564 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:02.747651 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:02.748186 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:38:02.748202 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:38:02.749759 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:38:02.751731 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:02.751739 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:02.751746 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:02.751975 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:02.752363 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:02.752365 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:02.752367 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:02.752480 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:02.753239 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:38:02.753245 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:38:02.753364 (+ 119us) client_negotiation.cc:769] Sending connection context
0927 02:38:02.753523 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 02:38:02.753798 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":114,"threads_started":1}
I20260927 02:38:02.754604 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:02.741767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60240 (local address 127.0.98.62:42593)
0927 02:38:02.742271 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:02.742284 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:02.742334 (+ 50us) server_negotiation.cc:407] Connection header received
0927 02:38:02.743338 (+ 1004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:02.743345 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:02.743589 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:02.743821 (+ 232us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:02.744758 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:02.746094 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:02.747081 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:02.747442 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:02.747568 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:02.749904 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:02.749909 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:02.749918 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:02.749939 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:02.751611 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:02.752092 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:02.752098 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:02.752100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:02.752242 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:02.752574 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:02.752578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:02.752580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:02.753105 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:02.753254 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:02.753620 (+ 366us) server_negotiation.cc:299] Negotiation successful
0927 02:38:02.753890 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":106,"server-negotiator.queue_time_us":389,"spinlock_wait_cycles":30464,"thread_start_us":142,"threads_started":1}
I20260927 02:38:02.756922 23082 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34259 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:38:02.757609 23085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:02.757368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34259 (local address 127.0.0.1:55514)
0927 02:38:02.757477 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:02.757545 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 02:38:02.757854 23030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34259: connect: Connection refused (error 111)
I20260927 02:38:02.758569 23085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:02.758401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34259 (local address 127.0.0.1:55520)
0927 02:38:02.758481 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:02.758522 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 02:38:02.758679 23082 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:34259: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34259: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 02:38:02.761943 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34259
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42593,127.0.98.61:34259,127.0.98.60:36527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:38:02.807894 23082 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34259 } attempt: 1
I20260927 02:38:02.808718 23085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:02.808437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34259 (local address 127.0.0.1:55530)
0927 02:38:02.808580 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:02.808640 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 02:38:02.809382 23085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:02.809173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34259 (local address 127.0.0.1:55536)
0927 02:38:02.809270 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:02.809313 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 02:38:02.809490 23082 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:34259: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34259: connect: Connection refused (error 111)
W20260927 02:38:02.865029 23089 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 02:38:02.865255 23089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:38:02.865288 23089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:38:02.865345 23089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:38:02.869191 23089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:38:02.869238 23089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:38:02.869262 23089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:38:02.869293 23089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:38:02.869323 23089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:38:02.874229 23089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42593,127.0.98.61:34259,127.0.98.60:36527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:38:02.875092 23089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:38:02.875975 23089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:38:02.877650 23082 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34259 } attempt: 2
I20260927 02:38:02.878464 23085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:02.878164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34259 (local address 127.0.0.1:55544)
0927 02:38:02.878330 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:02.878390 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 02:38:02.879117 23085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:02.878917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34259 (local address 127.0.0.1:55560)
0927 02:38:02.879012 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:02.879053 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 02:38:02.879222 23082 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:34259: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34259: connect: Connection refused (error 111)
W20260927 02:38:02.884024 23094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:38:02.884032 23095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:38:02.884107 23097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:38:02.884389 23089 server_base.cc:1034] running on GCE node
I20260927 02:38:02.884915 23089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:38:02.885942 23089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:38:02.887105 23089 hybrid_clock.cc:648] HybridClock initialized: now 1790476682887089 us; error 31 us; skew 500 ppm
Sep 27 02:38:02 dist-test-slave-grlk krb5kdc[23001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:38:02.889580 23089 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 02:38:02.890513 23089 webserver.cc:460] Webserver started at http://127.0.98.61:45193/ using document root <none> and password file <none>
I20260927 02:38:02.891144 23089 fs_manager.cc:359] Metadata directory not provided
I20260927 02:38:02.891261 23089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:38:02.891484 23089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:38:02.893595 23089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/minicluster-data/master-1/data/instance:
uuid: "84620b2fb56e4cea976cde93d407185e"
format_stamp: "Formatted at 2026-09-27 02:38:02 on dist-test-slave-grlk"
server_key: "62b07da54b5607f441c4ab9f5a9b263a"
server_key_iv: "e8bbcabaf04f1ce40876c27d9c0f875a"
server_key_version: "encryptionkey@0"
I20260927 02:38:02.894212 23089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/minicluster-data/master-1/wal/instance:
uuid: "84620b2fb56e4cea976cde93d407185e"
format_stamp: "Formatted at 2026-09-27 02:38:02 on dist-test-slave-grlk"
server_key: "62b07da54b5607f441c4ab9f5a9b263a"
server_key_iv: "e8bbcabaf04f1ce40876c27d9c0f875a"
server_key_version: "encryptionkey@0"
I20260927 02:38:02.898610 23089 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 02:38:02.900758 23104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:02.901881 23089 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:38:02.901985 23089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/minicluster-data/master-1/wal
uuid: "84620b2fb56e4cea976cde93d407185e"
format_stamp: "Formatted at 2026-09-27 02:38:02 on dist-test-slave-grlk"
server_key: "62b07da54b5607f441c4ab9f5a9b263a"
server_key_iv: "e8bbcabaf04f1ce40876c27d9c0f875a"
server_key_version: "encryptionkey@0"
I20260927 02:38:02.902077 23089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:38:02.915529 23089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:38:02.918322 23089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:38:02.918485 23089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:38:02.924614 23089 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:34259
I20260927 02:38:02.924868 23156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:34259 every 8 connection(s)
I20260927 02:38:02.925532 23089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/minicluster-data/master-1/data/info.pb
I20260927 02:38:02.926785 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 23089
I20260927 02:38:02.926901 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/minicluster-data/master-1/wal/instance
I20260927 02:38:02.927201 392 external_mini_cluster.cc:1428] Setting key 489a578f617c2dde6bee81b570b10c10
I20260927 02:38:02.928525 23157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:38:02.932193 23157 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42593 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 02:38:02 dist-test-slave-grlk krb5kdc[23001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:38:02.945111 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:02.934006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60242 (local address 127.0.98.62:42593)
0927 02:38:02.934149 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:02.934153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:02.934869 (+ 716us) server_negotiation.cc:407] Connection header received
0927 02:38:02.935804 (+ 935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:02.935807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:02.935876 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:02.935946 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:02.937086 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:02.937603 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:02.938695 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:02.939041 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:02.939136 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:02.941768 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:02.941771 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:02.941773 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:02.941800 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:02.943080 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:02.943621 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:02.943623 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:02.943624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:02.943694 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:02.944006 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:02.944008 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:02.944009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:02.944169 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:02.944242 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:02.944781 (+ 539us) server_negotiation.cc:299] Negotiation successful
0927 02:38:02.944903 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 02:38:02.945901 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:02.934321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42593 (local address 127.0.0.1:60242)
0927 02:38:02.934745 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:02.934784 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:02.935595 (+ 811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:02.936124 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:02.936131 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:02.936490 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:38:02.936952 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:02.936972 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:02.937721 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:02.937724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:02.938581 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:02.938590 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:02.939205 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:02.939208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:02.939352 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:02.939939 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:38:02.939954 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:38:02.941600 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:38:02.943196 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:02.943203 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:02.943218 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:02.943531 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:02.943771 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:02.943774 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:02.943776 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:02.943935 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:02.944284 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:38:02.944290 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:38:02.944563 (+ 273us) client_negotiation.cc:769] Sending connection context
0927 02:38:02.944760 (+ 197us) client_negotiation.cc:240] Negotiation successful
0927 02:38:02.945063 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":109,"threads_started":1}
I20260927 02:38:02.947408 23157 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34259 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 02:38:02.950091 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
/tmp/dist-test-taskcH4HTG/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36527
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42593,127.0.98.61:34259,127.0.98.60:36527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 02:38:02 dist-test-slave-grlk krb5kdc[23001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 02:38:02.956346 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:02.947866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34259 (local address 127.0.0.1:55566)
0927 02:38:02.948274 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:02.948286 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:02.948358 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:02.948712 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:02.948713 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:02.948855 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:38:02.948974 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:02.948981 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:02.950078 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:02.950081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:02.950807 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:02.950817 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:02.951374 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:02.951378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:02.951501 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:02.951914 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:38:02.951924 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:38:02.953189 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:38:02.954825 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:02.954828 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:02.954830 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:02.955074 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:02.955315 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:02.955317 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:02.955319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:02.955409 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:02.955774 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:38:02.955776 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:38:02.955884 (+ 108us) client_negotiation.cc:769] Sending connection context
0927 02:38:02.955994 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 02:38:02.956117 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356}
I20260927 02:38:02.956357 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:02.947975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55566 (local address 127.0.98.61:34259)
0927 02:38:02.948283 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:02.948293 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:02.948331 (+ 38us) server_negotiation.cc:407] Connection header received
0927 02:38:02.948432 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:02.948437 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:02.948613 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:02.948716 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:02.949067 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:02.949975 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:02.950927 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:02.951274 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:02.951368 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:02.953305 (+ 1937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:02.953313 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:02.953323 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:02.953343 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:02.954723 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:02.955168 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:02.955173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:02.955174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:02.955241 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:02.955484 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:02.955486 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:02.955487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:02.955686 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:02.955781 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:02.956049 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 02:38:02.956167 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":225,"thread_start_us":138,"threads_started":1}
I20260927 02:38:02.958305 23157 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36527 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:38:02.958961 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:02.958702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36527 (local address 127.0.0.1:56636)
0927 02:38:02.958808 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:02.958898 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 02:38:02.959133 23105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36527: connect: Connection refused (error 111)
I20260927 02:38:02.959888 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:02.959675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36527 (local address 127.0.0.1:56648)
0927 02:38:02.959776 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:02.959824 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 02:38:02.960007 23157 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36527: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36527: connect: Connection refused (error 111)
I20260927 02:38:02.970381 23082 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34259 } attempt: 3
Sep 27 02:38:02 dist-test-slave-grlk krb5kdc[23001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 02:38:02.978976 23085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:02.970902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34259 (local address 127.0.0.1:55576)
0927 02:38:02.971103 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:02.971118 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:02.971216 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:02.971457 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:02.971460 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:02.971645 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:38:02.971769 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:02.971775 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:02.972481 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:02.972484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:02.973817 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:02.973829 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:02.974256 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:02.974259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:02.974365 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:02.974796 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:38:02.974810 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:38:02.976135 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:38:02.977653 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:02.977656 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:02.977658 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:02.977890 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:02.978101 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:02.978104 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:02.978105 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:02.978190 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:02.978496 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:38:02.978499 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:38:02.978590 (+ 91us) client_negotiation.cc:769] Sending connection context
0927 02:38:02.978689 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 02:38:02.978806 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260927 02:38:02.978982 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:02.970993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55576 (local address 127.0.98.61:34259)
0927 02:38:02.971107 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:02.971111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:02.971246 (+ 135us) server_negotiation.cc:407] Connection header received
0927 02:38:02.971295 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:02.971297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:02.971343 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:02.971421 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:02.971873 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:02.972378 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:02.973929 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:02.974160 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:02.974237 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:02.976258 (+ 2021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:02.976261 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:02.976263 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:02.976284 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:02.977561 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:02.977968 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:02.977970 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:02.977971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:02.978034 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:02.978258 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:02.978259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:02.978260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:02.978422 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:02.978491 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:02.978735 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 02:38:02.978830 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 02:38:02.980238 23082 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36527 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:38:02.980835 23085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:02.980641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36527 (local address 127.0.0.1:56662)
0927 02:38:02.980741 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:02.980786 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 02:38:02.981422 23085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:02.981219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36527 (local address 127.0.0.1:56666)
0927 02:38:02.981333 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:02.981377 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 02:38:02.981518 23082 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36527: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36527: connect: Connection refused (error 111)
I20260927 02:38:03.009239 23157 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36527 } attempt: 1
I20260927 02:38:03.010104 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:03.009782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36527 (local address 127.0.0.1:56682)
0927 02:38:03.009940 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:03.010013 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 02:38:03.010866 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:03.010645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36527 (local address 127.0.0.1:56684)
0927 02:38:03.010755 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:03.010807 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 02:38:03.011004 23157 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36527: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36527: connect: Connection refused (error 111)
I20260927 02:38:03.012653 23082 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36527 } attempt: 1
I20260927 02:38:03.013262 23085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:03.013034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36527 (local address 127.0.0.1:56690)
0927 02:38:03.013147 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:03.013197 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 02:38:03.013878 23085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:03.013694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36527 (local address 127.0.0.1:56704)
0927 02:38:03.013785 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:03.013827 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 02:38:03.014001 23082 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36527: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36527: connect: Connection refused (error 111)
W20260927 02:38:03.059051 23163 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 02:38:03.059312 23163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:38:03.059334 23163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:38:03.059376 23163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:38:03.063320 23163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:38:03.063372 23163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:38:03.063390 23163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:38:03.063405 23163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:38:03.063421 23163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:38:03.068548 23163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42593,127.0.98.61:34259,127.0.98.60:36527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:38:03.069727 23163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:38:03.071898 23163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:38:03.079150 23157 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36527 } attempt: 2
I20260927 02:38:03.080140 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:03.079710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36527 (local address 127.0.0.1:56716)
0927 02:38:03.079918 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:03.079984 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 02:38:03.080579 23169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:38:03.080585 23170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:38:03.080614 23172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:38:03.080794 23163 server_base.cc:1034] running on GCE node
I20260927 02:38:03.081414 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:03.081214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36527 (local address 127.0.0.1:56728)
0927 02:38:03.081311 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:03.081356 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 02:38:03.081530 23157 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36527: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36527: connect: Connection refused (error 111)
I20260927 02:38:03.081702 23163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:38:03.082752 23163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:38:03.083928 23163 hybrid_clock.cc:648] HybridClock initialized: now 1790476683083908 us; error 33 us; skew 500 ppm
Sep 27 02:38:03 dist-test-slave-grlk krb5kdc[23001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:38:03.086525 23163 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 02:38:03.087359 23163 webserver.cc:460] Webserver started at http://127.0.98.60:41001/ using document root <none> and password file <none>
I20260927 02:38:03.087877 23163 fs_manager.cc:359] Metadata directory not provided
I20260927 02:38:03.087939 23163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:38:03.088132 23163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:38:03.089130 23082 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36527 } attempt: 2
I20260927 02:38:03.089777 23085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:03.089538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36527 (local address 127.0.0.1:56742)
0927 02:38:03.089662 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:03.089710 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 02:38:03.090361 23085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:03.090172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36527 (local address 127.0.0.1:56744)
0927 02:38:03.090265 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:03.090306 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 02:38:03.090466 23082 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36527: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36527: connect: Connection refused (error 111)
I20260927 02:38:03.090253 23163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/minicluster-data/master-2/data/instance:
uuid: "e806ac82df09400185edf13b93d3e648"
format_stamp: "Formatted at 2026-09-27 02:38:03 on dist-test-slave-grlk"
server_key: "3601d455b3574a0009f2bcdd3f60b894"
server_key_iv: "3710116f32bd658f0cd688afe0e89cbd"
server_key_version: "encryptionkey@0"
I20260927 02:38:03.090767 23163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/minicluster-data/master-2/wal/instance:
uuid: "e806ac82df09400185edf13b93d3e648"
format_stamp: "Formatted at 2026-09-27 02:38:03 on dist-test-slave-grlk"
server_key: "3601d455b3574a0009f2bcdd3f60b894"
server_key_iv: "3710116f32bd658f0cd688afe0e89cbd"
server_key_version: "encryptionkey@0"
I20260927 02:38:03.094353 23163 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:38:03.096627 23179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:03.097802 23163 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:38:03.097919 23163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/minicluster-data/master-2/wal
uuid: "e806ac82df09400185edf13b93d3e648"
format_stamp: "Formatted at 2026-09-27 02:38:03 on dist-test-slave-grlk"
server_key: "3601d455b3574a0009f2bcdd3f60b894"
server_key_iv: "3710116f32bd658f0cd688afe0e89cbd"
server_key_version: "encryptionkey@0"
I20260927 02:38:03.098003 23163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:38:03.120913 23163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:38:03.123593 23163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:38:03.123770 23163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:38:03.130756 23163 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:36527
I20260927 02:38:03.130999 23231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:36527 every 8 connection(s)
I20260927 02:38:03.131688 23163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/minicluster-data/master-2/data/info.pb
I20260927 02:38:03.134406 23232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:38:03.136107 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu as pid 23163
I20260927 02:38:03.136227 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcH4HTG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476410933508-392-0/minicluster-data/master-2/wal/instance
I20260927 02:38:03.136500 392 external_mini_cluster.cc:1428] Setting key 1c2bfe7f997d602a23d896f7154a92be
I20260927 02:38:03.138551 23232 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42593 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:38:03 dist-test-slave-grlk krb5kdc[23001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 02:38:03 dist-test-slave-grlk krb5kdc[23001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 02:38:03.149858 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:03.140180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60258 (local address 127.0.98.62:42593)
0927 02:38:03.140299 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:03.140304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:03.140690 (+ 386us) server_negotiation.cc:407] Connection header received
0927 02:38:03.140820 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:03.140838 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:03.140890 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:03.141001 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:03.141650 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:03.142301 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:03.143551 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:03.143845 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:03.143937 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:03.146142 (+ 2205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:03.146146 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:03.146149 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:03.146177 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:03.147626 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:03.148162 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:03.148166 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:03.148167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:03.148255 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:03.148621 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:03.148623 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:03.148625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:03.148857 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:03.148965 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:03.149453 (+ 488us) server_negotiation.cc:299] Negotiation successful
0927 02:38:03.149560 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 02:38:03.150496 23236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:03.140103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42593 (local address 127.0.0.1:60258)
0927 02:38:03.140549 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:03.140600 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:03.140723 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:03.141049 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:03.141053 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:03.141352 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:38:03.141518 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:03.141527 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:03.142424 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:03.142427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:03.143446 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:03.143455 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:03.143993 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:03.143998 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:03.144082 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:03.144468 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:38:03.144487 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:38:03.146019 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:38:03.147733 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:03.147741 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:03.147748 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:03.148072 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:03.148356 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:03.148359 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:03.148361 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:03.148541 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:03.148981 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:38:03.148987 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:38:03.149235 (+ 248us) client_negotiation.cc:769] Sending connection context
0927 02:38:03.149391 (+ 156us) client_negotiation.cc:240] Negotiation successful
0927 02:38:03.149703 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":50,"spinlock_wait_cycles":5504,"thread_start_us":102,"threads_started":1}
I20260927 02:38:03.150497 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:03.138201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56748 (local address 127.0.98.60:36527)
0927 02:38:03.138626 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:03.138640 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:03.138674 (+ 34us) server_negotiation.cc:407] Connection header received
0927 02:38:03.139359 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:03.139378 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:03.139700 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:03.140008 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:03.140820 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:03.142042 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:03.143147 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:03.143638 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:03.143775 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:03.146554 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:03.146559 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:03.146568 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:03.146591 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:03.148162 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:03.148584 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:03.148589 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:03.148591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:03.148674 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:03.148903 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:03.148906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:03.148908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:03.149224 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:03.149368 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:03.149596 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 02:38:03.149755 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":293,"thread_start_us":135,"threads_started":1}
I20260927 02:38:03.152154 23232 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34259 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:38:03 dist-test-slave-grlk krb5kdc[23001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 02:38:03 dist-test-slave-grlk krb5kdc[23001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:38:03.159466 23236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:03.152552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34259 (local address 127.0.0.1:55592)
0927 02:38:03.152634 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:03.152647 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:03.152710 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:03.153110 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:03.153114 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:03.153260 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:38:03.153350 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:03.153356 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:03.154004 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:03.154006 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:03.154504 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:03.154511 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:03.154887 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:03.154890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:03.154990 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:03.155362 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:38:03.155372 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:38:03.156568 (+ 1196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:38:03.158111 (+ 1543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:03.158114 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:03.158115 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:03.158335 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:03.158578 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:03.158580 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:03.158581 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:03.158660 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:03.158995 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:38:03.158999 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:38:03.159090 (+ 91us) client_negotiation.cc:769] Sending connection context
0927 02:38:03.159197 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 02:38:03.159306 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260927 02:38:03.159474 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:03.152788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55592 (local address 127.0.98.61:34259)
0927 02:38:03.152891 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:03.152895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:03.152906 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:38:03.152962 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:03.152965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:03.153007 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:03.153069 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:03.153441 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:03.153916 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:03.154599 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:03.154797 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:03.154868 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:03.156694 (+ 1826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:03.156696 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:03.156698 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:03.156720 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:03.158004 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:03.158422 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:03.158425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:03.158426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:03.158494 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:03.158746 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:03.158748 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:03.158749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:03.158908 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:03.158981 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:03.159230 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 02:38:03.159323 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 02:38:03.160076 23232 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36527 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:38:03.161299 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:03.152496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60272 (local address 127.0.98.62:42593)
0927 02:38:03.152589 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:03.152593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:03.152604 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:38:03.152696 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:03.152699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:03.152735 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:03.152792 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:03.154020 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:03.154472 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:03.155333 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:03.155555 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:03.155628 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:03.157864 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:03.157868 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:03.157870 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:03.157894 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:03.159294 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:03.160057 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:03.160060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:03.160061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:03.160130 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:03.160393 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:03.160396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:03.160398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:03.160604 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:03.160761 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:03.161056 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 02:38:03.161159 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Sep 27 02:38:03 dist-test-slave-grlk krb5kdc[23001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 02:38:03 dist-test-slave-grlk krb5kdc[23001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 02:38:03.167891 23236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:03.160477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36527 (local address 127.0.0.1:56758)
0927 02:38:03.160574 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:03.160583 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:03.160635 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:03.160859 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:03.160862 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:03.161044 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:38:03.161118 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:03.161125 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:03.162026 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:03.162030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:03.162720 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:03.162728 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:03.163235 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:03.163238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:03.163315 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:03.163659 (+ 344us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:38:03.163671 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:38:03.164674 (+ 1003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:38:03.166477 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:03.166480 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:03.166482 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:03.166680 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:03.166932 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:03.166935 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:03.166937 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:03.167028 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:03.167435 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:38:03.167438 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:38:03.167537 (+ 99us) client_negotiation.cc:769] Sending connection context
0927 02:38:03.167652 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 02:38:03.167784 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 02:38:03.168246 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:03.160612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56758 (local address 127.0.98.60:36527)
0927 02:38:03.160705 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:03.160709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:03.160719 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:38:03.160743 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:03.160746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:03.160775 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:03.160841 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:03.161331 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:03.161932 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:03.162829 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:03.163145 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:03.163214 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:03.164783 (+ 1569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:03.164786 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:03.164788 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:03.164810 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:03.166364 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:03.166770 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:03.166773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:03.166774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:03.166844 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:03.167114 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:03.167117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:03.167118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:03.167327 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:03.167420 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:03.167749 (+ 329us) server_negotiation.cc:299] Negotiation successful
0927 02:38:03.167855 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 02:38:03.170394 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:03.162252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55602 (local address 127.0.98.61:34259)
0927 02:38:03.162359 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:03.162362 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:03.162375 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:38:03.162449 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:03.162452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:03.162488 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:03.162557 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:03.163069 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:03.163674 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:03.164353 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:03.164625 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:03.164708 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:03.167148 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:03.167151 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:03.167153 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:03.167176 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:03.168786 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:03.169321 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:03.169324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:03.169325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:03.169398 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:03.169688 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:03.169691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:03.169694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:03.169879 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:03.170034 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:03.170197 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 02:38:03.170279 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 02:38:03.171854 23232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e806ac82df09400185edf13b93d3e648: Bootstrap starting.
I20260927 02:38:03.172670 23157 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36527 } attempt: 3
I20260927 02:38:03.173381 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:38:03.174286 23232 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e806ac82df09400185edf13b93d3e648: Neither blocks nor log segments found. Creating new log.
I20260927 02:38:03.175125 23232 log.cc:824] T 00000000000000000000000000000000 P e806ac82df09400185edf13b93d3e648: Log is configured to *not* fsync() on all Append() calls
I20260927 02:38:03.174628 23056 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "475e5e78e82f4710a3cb95975c35d7e6"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60272
I20260927 02:38:03.177328 23232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e806ac82df09400185edf13b93d3e648: No bootstrap required, opened a new log
Sep 27 02:38:03 dist-test-slave-grlk krb5kdc[23001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 02:38:03.181031 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:03.173401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56774 (local address 127.0.98.60:36527)
0927 02:38:03.173510 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:03.173513 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:03.173525 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:38:03.173554 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:03.173557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:03.173591 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:03.174057 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:03.174135 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:03.174729 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:03.175648 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:03.175950 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:03.176050 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:03.178091 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:03.178095 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:03.178098 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:03.178121 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:03.179436 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:03.179855 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:03.179858 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:03.179860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:03.179937 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:03.180190 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:03.180193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:03.180195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:03.180421 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:03.180506 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:03.180750 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 02:38:03.180853 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 02:38:03.181032 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:03.173087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36527 (local address 127.0.0.1:56774)
0927 02:38:03.173203 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:03.173218 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:03.173288 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:03.173700 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:03.173703 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:03.173863 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:38:03.173959 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:03.173965 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:03.174836 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:03.174839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:03.175516 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:03.175527 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:03.176060 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:03.176064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:03.176145 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:03.176564 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:38:03.176577 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:38:03.177947 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:38:03.179536 (+ 1589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:03.179539 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:03.179542 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:03.179768 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:03.180015 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:03.180018 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:03.180021 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:03.180107 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:03.180506 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:38:03.180508 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:38:03.180600 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 02:38:03.180704 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 02:38:03.180842 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 02:38:03.181095 23232 raft_consensus.cc:348] T 00000000000000000000000000000000 P e806ac82df09400185edf13b93d3e648 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "475e5e78e82f4710a3cb95975c35d7e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42593 } } peers { permanent_uuid: "84620b2fb56e4cea976cde93d407185e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34259 } } peers { permanent_uuid: "e806ac82df09400185edf13b93d3e648" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36527 } }
I20260927 02:38:03.181326 23232 raft_consensus.cc:729] T 00000000000000000000000000000000 P e806ac82df09400185edf13b93d3e648 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e806ac82df09400185edf13b93d3e648, State: Initialized, Role: FOLLOWER
I20260927 02:38:03.181612 23232 consensus_queue.cc:260] T 00000000000000000000000000000000 P e806ac82df09400185edf13b93d3e648 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "475e5e78e82f4710a3cb95975c35d7e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42593 } } peers { permanent_uuid: "84620b2fb56e4cea976cde93d407185e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34259 } } peers { permanent_uuid: "e806ac82df09400185edf13b93d3e648" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36527 } }
I20260927 02:38:03.182075 23238 sys_catalog.cc:455] T 00000000000000000000000000000000 P e806ac82df09400185edf13b93d3e648 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "475e5e78e82f4710a3cb95975c35d7e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42593 } } peers { permanent_uuid: "84620b2fb56e4cea976cde93d407185e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34259 } } peers { permanent_uuid: "e806ac82df09400185edf13b93d3e648" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36527 } } }
I20260927 02:38:03.182274 23238 sys_catalog.cc:458] T 00000000000000000000000000000000 P e806ac82df09400185edf13b93d3e648 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:38:03.182889 23232 sys_catalog.cc:564] T 00000000000000000000000000000000 P e806ac82df09400185edf13b93d3e648 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:38:03.183701 23157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84620b2fb56e4cea976cde93d407185e: Bootstrap starting.
I20260927 02:38:03.186362 23157 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84620b2fb56e4cea976cde93d407185e: Neither blocks nor log segments found. Creating new log.
I20260927 02:38:03.187189 23157 log.cc:824] T 00000000000000000000000000000000 P 84620b2fb56e4cea976cde93d407185e: Log is configured to *not* fsync() on all Append() calls
W20260927 02:38:03.187886 23249 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e806ac82df09400185edf13b93d3e648: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:38:03.187958 23249 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:38:03.189353 23157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84620b2fb56e4cea976cde93d407185e: No bootstrap required, opened a new log
I20260927 02:38:03.189548 23082 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36527 } attempt: 3
I20260927 02:38:03.186657 23163 master_runner.cc:186] Error getting master registration for 127.0.98.62:42593: OK, instance_id { permanent_uuid: "475e5e78e82f4710a3cb95975c35d7e6" instance_seqno: 1790476682730632 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 02:38:03.190199 23163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 02:38:03.193351 23157 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84620b2fb56e4cea976cde93d407185e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "475e5e78e82f4710a3cb95975c35d7e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42593 } } peers { permanent_uuid: "84620b2fb56e4cea976cde93d407185e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34259 } } peers { permanent_uuid: "e806ac82df09400185edf13b93d3e648" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36527 } }
I20260927 02:38:03.193584 23157 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84620b2fb56e4cea976cde93d407185e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84620b2fb56e4cea976cde93d407185e, State: Initialized, Role: FOLLOWER
I20260927 02:38:03.193979 23157 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84620b2fb56e4cea976cde93d407185e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "475e5e78e82f4710a3cb95975c35d7e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42593 } } peers { permanent_uuid: "84620b2fb56e4cea976cde93d407185e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34259 } } peers { permanent_uuid: "e806ac82df09400185edf13b93d3e648" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36527 } }
I20260927 02:38:03.194449 23252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84620b2fb56e4cea976cde93d407185e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "475e5e78e82f4710a3cb95975c35d7e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42593 } } peers { permanent_uuid: "84620b2fb56e4cea976cde93d407185e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34259 } } peers { permanent_uuid: "e806ac82df09400185edf13b93d3e648" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36527 } } }
I20260927 02:38:03.194574 23252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84620b2fb56e4cea976cde93d407185e [sys.catalog]: This master's current role is: FOLLOWER
Sep 27 02:38:03 dist-test-slave-grlk krb5kdc[23001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 02:38:03.195279 23157 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84620b2fb56e4cea976cde93d407185e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:38:03.199098 23085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:03.189919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36527 (local address 127.0.0.1:56788)
0927 02:38:03.190181 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:03.190197 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:03.190291 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:03.190758 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:03.190762 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:03.190953 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:38:03.191065 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:03.191073 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:03.192049 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:03.192053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:03.192882 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:03.192894 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:03.193524 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:03.193529 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:03.193622 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:03.194127 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:38:03.194143 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:38:03.195624 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:38:03.197527 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:03.197531 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:03.197533 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:03.197785 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:03.198078 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:03.198081 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:03.198083 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:03.198173 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:03.198598 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:38:03.198602 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:38:03.198699 (+ 97us) client_negotiation.cc:769] Sending connection context
0927 02:38:03.198807 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 02:38:03.198940 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182}
I20260927 02:38:03.199653 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:03.190430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56788 (local address 127.0.98.60:36527)
0927 02:38:03.190557 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:03.190561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:03.190574 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:38:03.190606 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:03.190608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:03.190647 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:03.190728 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:03.191196 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:03.191940 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:03.193047 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:03.193394 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:03.193521 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:03.195799 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:03.195803 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:03.195805 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:03.195829 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:03.197410 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:03.197895 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:03.197898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:03.197900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:03.197981 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:03.198267 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:03.198270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:03.198272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:03.198465 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:03.198552 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:03.199405 (+ 853us) server_negotiation.cc:299] Negotiation successful
0927 02:38:03.199519 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 02:38:03.199056 23089 master_runner.cc:186] Error getting master registration for 127.0.98.62:42593: OK, instance_id { permanent_uuid: "475e5e78e82f4710a3cb95975c35d7e6" instance_seqno: 1790476682730632 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 02:38:03.202756 23082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 475e5e78e82f4710a3cb95975c35d7e6: Bootstrap starting.
I20260927 02:38:03.202837 23089 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 02:38:03.204811 23082 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 475e5e78e82f4710a3cb95975c35d7e6: Neither blocks nor log segments found. Creating new log.
I20260927 02:38:03.205610 23082 log.cc:824] T 00000000000000000000000000000000 P 475e5e78e82f4710a3cb95975c35d7e6: Log is configured to *not* fsync() on all Append() calls
W20260927 02:38:03.206782 23263 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 84620b2fb56e4cea976cde93d407185e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:38:03.206848 23263 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:38:03.208232 23082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 475e5e78e82f4710a3cb95975c35d7e6: No bootstrap required, opened a new log
I20260927 02:38:03.212471 23082 raft_consensus.cc:348] T 00000000000000000000000000000000 P 475e5e78e82f4710a3cb95975c35d7e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "475e5e78e82f4710a3cb95975c35d7e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42593 } } peers { permanent_uuid: "84620b2fb56e4cea976cde93d407185e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34259 } } peers { permanent_uuid: "e806ac82df09400185edf13b93d3e648" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36527 } }
I20260927 02:38:03.212769 23082 raft_consensus.cc:729] T 00000000000000000000000000000000 P 475e5e78e82f4710a3cb95975c35d7e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 475e5e78e82f4710a3cb95975c35d7e6, State: Initialized, Role: FOLLOWER
I20260927 02:38:03.213259 23082 consensus_queue.cc:260] T 00000000000000000000000000000000 P 475e5e78e82f4710a3cb95975c35d7e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "475e5e78e82f4710a3cb95975c35d7e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42593 } } peers { permanent_uuid: "84620b2fb56e4cea976cde93d407185e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34259 } } peers { permanent_uuid: "e806ac82df09400185edf13b93d3e648" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36527 } }
I20260927 02:38:03.213858 23266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 475e5e78e82f4710a3cb95975c35d7e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "475e5e78e82f4710a3cb95975c35d7e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42593 } } peers { permanent_uuid: "84620b2fb56e4cea976cde93d407185e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34259 } } peers { permanent_uuid: "e806ac82df09400185edf13b93d3e648" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36527 } } }
I20260927 02:38:03.214053 23266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 475e5e78e82f4710a3cb95975c35d7e6 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:38:03.215167 23056 raft_consensus.cc:482] T 00000000000000000000000000000000 P 475e5e78e82f4710a3cb95975c35d7e6 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 02:38:03.215276 23056 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 475e5e78e82f4710a3cb95975c35d7e6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:38:03.216465 23056 raft_consensus.cc:504] T 00000000000000000000000000000000 P 475e5e78e82f4710a3cb95975c35d7e6 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "475e5e78e82f4710a3cb95975c35d7e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42593 } } peers { permanent_uuid: "84620b2fb56e4cea976cde93d407185e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34259 } } peers { permanent_uuid: "e806ac82df09400185edf13b93d3e648" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36527 } }
I20260927 02:38:03.217386 23056 leader_election.cc:288] T 00000000000000000000000000000000 P 475e5e78e82f4710a3cb95975c35d7e6 [CANDIDATE]: Term 1 election: Requested vote from peers 84620b2fb56e4cea976cde93d407185e (127.0.98.61:34259), e806ac82df09400185edf13b93d3e648 (127.0.98.60:36527)
I20260927 02:38:03.215111 23082 sys_catalog.cc:564] T 00000000000000000000000000000000 P 475e5e78e82f4710a3cb95975c35d7e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:38:03.218250 23131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "475e5e78e82f4710a3cb95975c35d7e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "84620b2fb56e4cea976cde93d407185e"
I20260927 02:38:03.218497 23131 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84620b2fb56e4cea976cde93d407185e [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:38:03.219784 23131 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 84620b2fb56e4cea976cde93d407185e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 475e5e78e82f4710a3cb95975c35d7e6 in term 1.
I20260927 02:38:03.219784 23206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "475e5e78e82f4710a3cb95975c35d7e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e806ac82df09400185edf13b93d3e648"
I20260927 02:38:03.220023 23206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e806ac82df09400185edf13b93d3e648 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:38:03.220222 23030 leader_election.cc:302] T 00000000000000000000000000000000 P 475e5e78e82f4710a3cb95975c35d7e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 475e5e78e82f4710a3cb95975c35d7e6, 84620b2fb56e4cea976cde93d407185e; no voters:
I20260927 02:38:03.220461 23266 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 475e5e78e82f4710a3cb95975c35d7e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:38:03.220746 23266 raft_consensus.cc:686] T 00000000000000000000000000000000 P 475e5e78e82f4710a3cb95975c35d7e6 [term 1 LEADER]: Becoming Leader. State: Replica: 475e5e78e82f4710a3cb95975c35d7e6, State: Running, Role: LEADER
I20260927 02:38:03.221136 23266 consensus_queue.cc:237] T 00000000000000000000000000000000 P 475e5e78e82f4710a3cb95975c35d7e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "475e5e78e82f4710a3cb95975c35d7e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42593 } } peers { permanent_uuid: "84620b2fb56e4cea976cde93d407185e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34259 } } peers { permanent_uuid: "e806ac82df09400185edf13b93d3e648" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36527 } }
I20260927 02:38:03.221382 23206 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e806ac82df09400185edf13b93d3e648 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 475e5e78e82f4710a3cb95975c35d7e6 in term 1.
I20260927 02:38:03.223963 23282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 475e5e78e82f4710a3cb95975c35d7e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 475e5e78e82f4710a3cb95975c35d7e6. Latest consensus state: current_term: 1 leader_uuid: "475e5e78e82f4710a3cb95975c35d7e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "475e5e78e82f4710a3cb95975c35d7e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42593 } } peers { permanent_uuid: "84620b2fb56e4cea976cde93d407185e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34259 } } peers { permanent_uuid: "e806ac82df09400185edf13b93d3e648" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36527 } } }
I20260927 02:38:03.224085 23282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 475e5e78e82f4710a3cb95975c35d7e6 [sys.catalog]: This master's current role is: LEADER
I20260927 02:38:03.224478 23284 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260927 02:38:03.227078 23035 tablet.cc:2369] T 00000000000000000000000000000000 P 475e5e78e82f4710a3cb95975c35d7e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:38:03.227690 23284 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:38:03.240139 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:03.225263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60286 (local address 127.0.98.62:42593)
0927 02:38:03.225367 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:03.225371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:03.229747 (+ 4376us) server_negotiation.cc:407] Connection header received
0927 02:38:03.229861 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:03.229864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:03.229912 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:03.229985 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:03.230924 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:03.232139 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:03.233897 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:03.234478 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:03.234610 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:03.236627 (+ 2017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:03.236632 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:03.236635 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:03.236662 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:03.238354 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:03.238828 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:03.238831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:03.238833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:03.238916 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:03.239201 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:03.239204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:03.239206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:03.239415 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:03.239512 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:03.239852 (+ 340us) server_negotiation.cc:299] Negotiation successful
0927 02:38:03.239974 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 02:38:03.243891 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:03.225916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55614 (local address 127.0.98.61:34259)
0927 02:38:03.226033 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:03.226037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:03.230044 (+ 4007us) server_negotiation.cc:407] Connection header received
0927 02:38:03.231590 (+ 1546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:03.231594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:03.231647 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:03.231704 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:03.234590 (+ 2886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:03.235486 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:03.237194 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:03.237531 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:03.237600 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:03.240493 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:03.240497 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:03.240500 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:03.240527 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:03.242103 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:03.242595 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:03.242598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:03.242600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:03.242691 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:03.242990 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:03.242993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:03.242995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:03.243196 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:03.243296 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:03.243604 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 02:38:03.243720 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 02:38:03.245941 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:03.226493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56794 (local address 127.0.98.60:36527)
0927 02:38:03.226597 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:03.226600 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:03.230250 (+ 3650us) server_negotiation.cc:407] Connection header received
0927 02:38:03.230509 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:03.230512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:03.230558 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:03.230610 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:03.232667 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:03.233383 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:03.236609 (+ 3226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:03.236995 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:03.237066 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:03.242616 (+ 5550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:03.242620 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:03.242622 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:03.242652 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:03.244173 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:03.244612 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:03.244615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:03.244617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:03.244706 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:03.245039 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:03.245043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:03.245044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:03.245255 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:03.245347 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:03.245658 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 02:38:03.245773 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 02:38:03.257071 23206 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e806ac82df09400185edf13b93d3e648 [term 1 FOLLOWER]: Refusing update from remote peer 475e5e78e82f4710a3cb95975c35d7e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:38:03.257766 23282 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 475e5e78e82f4710a3cb95975c35d7e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e806ac82df09400185edf13b93d3e648" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:38:03.257860 23131 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 84620b2fb56e4cea976cde93d407185e [term 1 FOLLOWER]: Refusing update from remote peer 475e5e78e82f4710a3cb95975c35d7e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:38:03.258673 23282 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 475e5e78e82f4710a3cb95975c35d7e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84620b2fb56e4cea976cde93d407185e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:38:03.263013 23252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84620b2fb56e4cea976cde93d407185e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 475e5e78e82f4710a3cb95975c35d7e6. Latest consensus state: current_term: 1 leader_uuid: "475e5e78e82f4710a3cb95975c35d7e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "475e5e78e82f4710a3cb95975c35d7e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42593 } } peers { permanent_uuid: "84620b2fb56e4cea976cde93d407185e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34259 } } peers { permanent_uuid: "e806ac82df09400185edf13b93d3e648" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36527 } } }
I20260927 02:38:03.263167 23252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84620b2fb56e4cea976cde93d407185e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:38:03.267517 23238 sys_catalog.cc:455] T 00000000000000000000000000000000 P e806ac82df09400185edf13b93d3e648 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 475e5e78e82f4710a3cb95975c35d7e6. Latest consensus state: current_term: 1 leader_uuid: "475e5e78e82f4710a3cb95975c35d7e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "475e5e78e82f4710a3cb95975c35d7e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42593 } } peers { permanent_uuid: "84620b2fb56e4cea976cde93d407185e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34259 } } peers { permanent_uuid: "e806ac82df09400185edf13b93d3e648" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36527 } } }
I20260927 02:38:03.267658 23238 sys_catalog.cc:458] T 00000000000000000000000000000000 P e806ac82df09400185edf13b93d3e648 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:38:03.271087 23238 sys_catalog.cc:455] T 00000000000000000000000000000000 P e806ac82df09400185edf13b93d3e648 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "475e5e78e82f4710a3cb95975c35d7e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "475e5e78e82f4710a3cb95975c35d7e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42593 } } peers { permanent_uuid: "84620b2fb56e4cea976cde93d407185e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34259 } } peers { permanent_uuid: "e806ac82df09400185edf13b93d3e648" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36527 } } }
I20260927 02:38:03.271227 23238 sys_catalog.cc:458] T 00000000000000000000000000000000 P e806ac82df09400185edf13b93d3e648 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:38:03.273636 23292 mvcc.cc:205] Tried to move back new op lower bound from 7333792494611607552 to 7333792494477545472. Current Snapshot: MvccSnapshot[applied={T|T < 7333792494611607552}]
I20260927 02:38:03.274683 23252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84620b2fb56e4cea976cde93d407185e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "475e5e78e82f4710a3cb95975c35d7e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "475e5e78e82f4710a3cb95975c35d7e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42593 } } peers { permanent_uuid: "84620b2fb56e4cea976cde93d407185e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34259 } } peers { permanent_uuid: "e806ac82df09400185edf13b93d3e648" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36527 } } }
I20260927 02:38:03.274806 23252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84620b2fb56e4cea976cde93d407185e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:38:03.275344 23290 mvcc.cc:205] Tried to move back new op lower bound from 7333792494611607552 to 7333792494477545472. Current Snapshot: MvccSnapshot[applied={T|T < 7333792494611607552}]
I20260927 02:38:03.275498 23289 mvcc.cc:205] Tried to move back new op lower bound from 7333792494611607552 to 7333792494477545472. Current Snapshot: MvccSnapshot[applied={T|T < 7333792494611607552}]
I20260927 02:38:03.276124 23266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 475e5e78e82f4710a3cb95975c35d7e6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "475e5e78e82f4710a3cb95975c35d7e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "475e5e78e82f4710a3cb95975c35d7e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42593 } } peers { permanent_uuid: "84620b2fb56e4cea976cde93d407185e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34259 } } peers { permanent_uuid: "e806ac82df09400185edf13b93d3e648" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36527 } } }
I20260927 02:38:03.276253 23266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 475e5e78e82f4710a3cb95975c35d7e6 [sys.catalog]: This master's current role is: LEADER
I20260927 02:38:03.276360 23266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 475e5e78e82f4710a3cb95975c35d7e6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "475e5e78e82f4710a3cb95975c35d7e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "475e5e78e82f4710a3cb95975c35d7e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42593 } } peers { permanent_uuid: "84620b2fb56e4cea976cde93d407185e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34259 } } peers { permanent_uuid: "e806ac82df09400185edf13b93d3e648" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36527 } } }
I20260927 02:38:03.276436 23266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 475e5e78e82f4710a3cb95975c35d7e6 [sys.catalog]: This master's current role is: LEADER
I20260927 02:38:03.282925 23284 catalog_manager.cc:1347] Generated new cluster ID: 236dabf9880547f7b12661b176d84d18
I20260927 02:38:03.282989 23284 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:38:03.311676 23284 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:38:03.313138 23284 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:38:03.338379 23284 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 475e5e78e82f4710a3cb95975c35d7e6: Generated new TSK 0
I20260927 02:38:03.339296 23284 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 02:38:03.369947 23086 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:38:03.367110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60298 (local address 127.0.98.62:42593)
0927 02:38:03.367247 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:03.367251 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:03.368874 (+ 1623us) server_negotiation.cc:407] Connection header received
0927 02:38:03.369113 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:03.369117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:03.369169 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:03.369231 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:03.369813 (+ 582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60298: BlockingRecv error: recv got EOF from 127.0.0.1:60298 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 02:38:03.371371 23162 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:38:03.367795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55626 (local address 127.0.98.61:34259)
0927 02:38:03.367902 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:03.367906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:03.370554 (+ 2648us) server_negotiation.cc:407] Connection header received
0927 02:38:03.370707 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:03.370711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:03.370759 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:03.370806 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:03.371256 (+ 450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55626: BlockingRecv error: recv got EOF from 127.0.0.1:55626 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260927 02:38:03.372923 23235 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:38:03.368407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56800 (local address 127.0.98.60:36527)
0927 02:38:03.368540 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:03.368544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:03.371880 (+ 3336us) server_negotiation.cc:407] Connection header received
0927 02:38:03.372239 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:03.372242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:03.372291 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:03.372339 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:03.372812 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56800: BlockingRecv error: recv got EOF from 127.0.0.1:56800 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260927 02:38:03.378516 23162 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:38:03.376972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55634 (local address 127.0.98.61:34259)
0927 02:38:03.377088 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:03.377093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:03.377540 (+ 447us) server_negotiation.cc:407] Connection header received
0927 02:38:03.377751 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:03.377754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:03.377829 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:03.377881 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:03.378409 (+ 528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55634: BlockingRecv error: recv got EOF from 127.0.0.1:55634 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260927 02:38:03.384055 23162 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:38:03.382315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55650 (local address 127.0.98.61:34259)
0927 02:38:03.382468 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:03.382484 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:03.383010 (+ 526us) server_negotiation.cc:407] Connection header received
0927 02:38:03.383196 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:03.383200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:03.383249 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:03.383395 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:03.383933 (+ 538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55650: BlockingRecv error: recv got EOF from 127.0.0.1:55650 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260927 02:38:03.389804 23162 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:38:03.388312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55660 (local address 127.0.98.61:34259)
0927 02:38:03.388422 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:03.388426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:03.388887 (+ 461us) server_negotiation.cc:407] Connection header received
0927 02:38:03.389101 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:03.389104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:03.389153 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:03.389203 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:03.389702 (+ 499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55660: BlockingRecv error: recv got EOF from 127.0.0.1:55660 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260927 02:38:03.397992 23162 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:38:03.396329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55662 (local address 127.0.98.61:34259)
0927 02:38:03.396483 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:03.396488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:03.397004 (+ 516us) server_negotiation.cc:407] Connection header received
0927 02:38:03.397232 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:03.397236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:03.397289 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:03.397340 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:03.397876 (+ 536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55662: BlockingRecv error: recv got EOF from 127.0.0.1:55662 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260927 02:38:03.409741 23162 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:38:03.408419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55670 (local address 127.0.98.61:34259)
0927 02:38:03.408535 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:03.408540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:03.408819 (+ 279us) server_negotiation.cc:407] Connection header received
0927 02:38:03.408985 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:03.408989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:03.409054 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:03.409148 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:03.409619 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55670: BlockingRecv error: recv got EOF from 127.0.0.1:55670 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260927 02:38:03.429011 23162 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:38:03.427760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55674 (local address 127.0.98.61:34259)
0927 02:38:03.427932 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:03.427937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:03.428124 (+ 187us) server_negotiation.cc:407] Connection header received
0927 02:38:03.428266 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:03.428269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:03.428329 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:03.428436 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:03.428848 (+ 412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55674: BlockingRecv error: recv got EOF from 127.0.0.1:55674 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 02:38:03.464484 23162 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:38:03.463300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55690 (local address 127.0.98.61:34259)
0927 02:38:03.463462 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:03.463468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:03.463565 (+ 97us) server_negotiation.cc:407] Connection header received
0927 02:38:03.463733 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:03.463736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:03.463797 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:03.463877 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:03.464355 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55690: BlockingRecv error: recv got EOF from 127.0.0.1:55690 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260927 02:38:03.531594 23162 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:38:03.530489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55696 (local address 127.0.98.61:34259)
0927 02:38:03.530641 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:03.530645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:03.530755 (+ 110us) server_negotiation.cc:407] Connection header received
0927 02:38:03.530903 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:03.530906 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:03.530956 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:03.531025 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:03.531473 (+ 448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55696: BlockingRecv error: recv got EOF from 127.0.0.1:55696 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260927 02:38:03.662711 23162 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:38:03.661576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55702 (local address 127.0.98.61:34259)
0927 02:38:03.661733 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:03.661738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:03.661848 (+ 110us) server_negotiation.cc:407] Connection header received
0927 02:38:03.661997 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:03.662000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:03.662051 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:03.662123 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:03.662586 (+ 463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55702: BlockingRecv error: recv got EOF from 127.0.0.1:55702 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260927 02:38:03.921885 23162 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:38:03.920787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55706 (local address 127.0.98.61:34259)
0927 02:38:03.920971 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:03.920976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:03.921055 (+ 79us) server_negotiation.cc:407] Connection header received
0927 02:38:03.921197 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:03.921200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:03.921247 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:03.921317 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:03.921777 (+ 460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55706: BlockingRecv error: recv got EOF from 127.0.0.1:55706 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 02:38:04.189215 23249 catalog_manager.cc:1259] Loaded cluster ID: 236dabf9880547f7b12661b176d84d18
I20260927 02:38:04.189296 23249 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e806ac82df09400185edf13b93d3e648: loading cluster ID for follower catalog manager: success
I20260927 02:38:04.190908 23249 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e806ac82df09400185edf13b93d3e648: acquiring CA information for follower catalog manager: success
I20260927 02:38:04.192027 23249 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e806ac82df09400185edf13b93d3e648: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 02:38:04.207988 23263 catalog_manager.cc:1259] Loaded cluster ID: 236dabf9880547f7b12661b176d84d18
I20260927 02:38:04.208058 23263 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 84620b2fb56e4cea976cde93d407185e: loading cluster ID for follower catalog manager: success
I20260927 02:38:04.209671 23263 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 84620b2fb56e4cea976cde93d407185e: acquiring CA information for follower catalog manager: success
I20260927 02:38:04.210742 23263 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 84620b2fb56e4cea976cde93d407185e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 02:38:04.439742 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:04.436017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55720 (local address 127.0.98.61:34259)
0927 02:38:04.436291 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:04.436296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:04.436312 (+ 16us) server_negotiation.cc:407] Connection header received
0927 02:38:04.436371 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:04.436375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:04.436495 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:04.436591 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:38:04.436851 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:04.437436 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:04.438301 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:04.438516 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:04.438600 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:04.438902 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:38:04.439332 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:38:04.439435 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:04.439535 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 02:38:04.439589 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":89,"threads_started":1}
I20260927 02:38:04.445705 23369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:04.442401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56814 (local address 127.0.98.60:36527)
0927 02:38:04.442619 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:04.442623 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:04.442657 (+ 34us) server_negotiation.cc:407] Connection header received
0927 02:38:04.442731 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:04.442733 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:04.442838 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:04.442932 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:38:04.443198 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:04.443713 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:04.444460 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:04.444689 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:04.444764 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:04.444992 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:38:04.445384 (+ 392us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:38:04.445473 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:04.445523 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 02:38:04.445570 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":63,"threads_started":1}
I20260927 02:38:04.452052 23375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:04.448247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60314 (local address 127.0.98.62:42593)
0927 02:38:04.448462 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:04.448466 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:04.448481 (+ 15us) server_negotiation.cc:407] Connection header received
0927 02:38:04.448567 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:04.448570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:04.448701 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:04.448819 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:38:04.449083 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:04.449801 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:04.450602 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:04.450940 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:04.451034 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:04.451409 (+ 375us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:38:04.451637 (+ 228us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:38:04.451715 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:04.451848 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 02:38:04.451904 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":58,"threads_started":1}
I20260927 02:38:04.452472 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:04.448533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55724 (local address 127.0.98.61:34259)
0927 02:38:04.448653 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:04.448657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:04.449047 (+ 390us) server_negotiation.cc:407] Connection header received
0927 02:38:04.449192 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:04.449195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:04.449245 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:04.449312 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:38:04.449516 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:04.450229 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:04.451370 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:04.451705 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:04.451801 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:04.452028 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:38:04.452103 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:38:04.452163 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:04.452302 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 02:38:04.452355 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 02:38:04.452489 23369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:04.448863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56828 (local address 127.0.98.60:36527)
0927 02:38:04.448988 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:04.448992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:04.449105 (+ 113us) server_negotiation.cc:407] Connection header received
0927 02:38:04.449286 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:04.449289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:04.449338 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:04.449402 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:38:04.449716 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:04.450417 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:04.451381 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:04.451705 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:04.451784 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:04.452079 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:38:04.452149 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:38:04.452227 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:04.452346 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 02:38:04.452390 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 02:38:04.455278 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 23014
I20260927 02:38:04.460268 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 23089
I20260927 02:38:04.464882 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcH4HTG/build/debug/bin/kudu with pid 23163
2026-09-27T02:38:04Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2996 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5963 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (273523 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST